1. EXECUTIVE SUMMARY: 

Applicant Information: This proposal is being submitted as a collaborative effort between 
Coordinated Care Services, Inc. (CCSI) and New York University, McSilver Institute for Poverty 
Policy and Research (NYU-McSilver). CCSI is the lead agency for the proposal. For CCSI, the 
Principal Investigator is John F. Crilly, PhD, MPH, MSW. Dr. Crilly has nearly 30 years of 
experience in the public sector (local, state, federal) and academia in New York State and 
Louisiana. He has held clinical, administrative, and research positions in his career which has 
centered on improving quality within organizations and the practical use of health informatics 
and electronic communication. Andrew F. Cleek, PsyD, NYU-McSilver, is the project Co- 
Investigator. Dr. Cleek has also worked in the public, private, and academic sectors, particularly 
with agencies involved in providing support and services to individuals in poverty, particularly 
children. He currently leads a major New York State (NYS) initiative to provide technical 
assistance to mental health centers to improve their approach to interventions and practice 
and their financial operations. Both investigators lead teams of highly competent individuals in 
the fields of education, research, and evaluation and both have had experience developing 
software applications and tools to help provide decision support for particular target groups. 

For example. Dr. Crilly created an award-winning innovative decision support software model in 
2012 using the electronic health record of the US Dept, of Veterans Affairs. Dr. Cleek has 
developed client-facing apps for smartphones (e.g., for providing homework support for 
children in poverty situations in New York City) and is developing app software for countries in 
Africa. In doing so, they have been immersed in the process required to actually formulate the 
groundwork to develop the software - understanding the issues, understanding the needs of 
stakeholders and users, and gaining expert input from all before the build begins. This ensures 
a higher quality end product and a more efficient software build. 

Information about Proposed Partnerships: CCSI and NYU-McSilver are currently collaborators 
on a large project for NYS. Described further below, it is the Clinical Technical Assistance Center 
(CTAC) project. Its purpose is to instill efficient clinical and financial practices in mental health 
clinics across NYS. NYU-McSilver (Cleek, PI) is the lead and includes CCSI and other partners. 
Brief Description of Decision Support Issues/Questions: We will focus on 2 areas identified in 
the RFP: A) "How can we assist workers to use the data collected on client (family) risks, 
strengths, and needs to target interventions (including specific evidence-based interventions)?" 
and B) "Which clients receiving behavioral services may not be receiving services at the 
appropriate level of care?" 

Brief Description of Proposed Solution: We are experienced in the entire development process 
of decision support tools, including software build. We know that the quality developed at the 
front-end of the process - the discovery of what is needed and determination of the predictive 
modeling - will have true pay-offs at the back-end where the software is built. Without 
sufficient attention paid to the front end, there is greater likelihood that the derived product 
will be of poor quality which directly impacts the ability to market that product to other DHS 
agencies, thus diminishing sustainability. We have therefore chosen to focus our efforts on 
providing a high quality front-end process and coordinating the integration of those results into 
the end product - confirming whether a software build is, in fact, necessary. 

Funding Request: We are requesting a total of $317,925 for Personnel and Associated Agency 
costs and $38,228 for Travel and Associated Costs for a total of $356,153. 
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2. NARRATIVE: 

1) Describe your organizations' history and experience conducting research, translating 
research into practice, developing/validating tools built on research, developing and 
implementing predictive models. 

This proposed project is part of an ongoing collaboration between Coordinated Care Services, 
Inc. and New York University's McSilver Institute on Poverty Policy and Research. Each is 
addressed separately below. 

A. Organization Histories 
Coordinated Care Services, Inc. 

Coordinated Care Services, Inc. (CCSI), the lead agency for this proposal, is a not-for-profit 
management services organization with demonstrated expertise in the areas of behavioral 
health and human services. Based in Rochester, NY, CCSI provides a broad array of services, 
including: performance management, program and system evaluation, project and contract 
management, data analysis, financial management and support services, training and practice 
transformation services, including cultural competence assessment and training. CCSI has more 
than 20 years of experience working with county departments across New York State (NYS) and, 
more recently, in the District of Columbia. Previous work has included collaborating on the 
design of evaluation and quality improvement projects, serving as independent evaluator for 
local and federally funded projects, conducting system needs assessments, developing 
performance management plans, assisting with Plan-Do-Study-Act continuous quality 
improvement cycles, and providing data analysis support to federally funded research projects 
based in university and community settings. 

New York University (NYU) - McSilver Institute for Poverty, Policy, and Research 

The NYU McSilver Institute was founded in 2007 with the purpose of better understanding 
poverty, particularly in New York City (NYC), and developing practices to address it. The 
Institute studies current poverty-related issues and programs, pilots interventions in human 
services agencies, and serves as a supportive resource and change agent for agencies working 
to address poverty. It does this by conducting, promoting, and disseminating interdisciplinary 
applied research to address root causes of, effects of, and responses to poverty. Drawing on 
the substantial intellectual and scholarly resources of NYU and located within the Silver School 
of Social Work, the Institute partners with agencies and communities to develop research 
projects and policy/advocacy recommendations that have short-and long-term social impact. 

NYU-McSilver recognizes the significant link between individuals, families, and 
communities and their external environments, as well as the interrelatedness of race and 
poverty. Within this important framework, NYU-McSilver sponsors and fosters: 

• Applied Research in diverse areas of direct social, psychological, legal, and community¬ 
wide endeavors designed to assist those living in or near-poverty 

• Development of Mutually Beneficial Partnerships between NYU-McSilver community- 
based agencies and organizations that are "on-the-ground" working with people living in 
poverty 

• Organizational Development efforts that encourage untapped strengths in agencies and 
communities through research and evaluation 
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• Policy Development and Dissemination grounded in evidence-based research focused 
on "theories of justice," the causes and consequences of poverty, and poverty 
amelioration 

• Locating Research and Related Projects in the larger field of poverty studies and the 
work of scholars and agencies all over the world. 

B. Experience conducting research, translating research into practice, developing/validating 
tools built on research, developing and implementing predictive models 

Examples of CCSI's and NYU-McSilver's work illustrating our ability to conduct research - and 
importantly, to translate that research into practices that help to improve service delivery and 
the associated outcomes, are summarized briefly in this section. 

Coordinated Care Services, Inc. 

1. New York Care Coordination Program (NYCCP) : Categories: Translating Research into 
Practice, Predictive Modeling. 

The NYCCP is a not-for-profit organization, with expertise in developing person- 
centered, recovery-focused systems of care in both urban and rural environments to improve 
access, quality, and cost-effectiveness of publicly funded behavioral health services. 

Established in 2002 as a collaboration of local county mental health departments, providers, 
and peers/family members, NYCCP includes 10 New York counties. Most recently NYCCP 
worked in partnership with Beacon Health Strategies and CCSI to serve as the Behavioral Health 
Organization (BHO) for the western region of NYS. NYCCP leads the same partnership with 
Beacon and CCSI to manage the implementation of Medicaid Health Homes (HH) in 22 upstate 
counties. 

CCSI has provided a full range of management supports to the NYCCP since its inception, 
including staffing key project functions (the NYCCP Project Director and support staff), fiscal 
management, and data analysis and reporting. Throughout its history, NYCCP's work has been 
guided by the ongoing collection, analysis, and use of performance data to improve practice. 
Examples include: 

• Development of a process to assess implementation and fidelity of person-centered 
practices by care managers. We developed results to be used as decision supports for 
designing new interventions which led to the development of a training curriculum and 
related tools used by providers across NYS (noted in Dulmus and Nisbet, 2013). 

• Analysis of Medicaid claims data to assess the extent to which care coordination services 
were resulting in reductions in the use of acute services and increases in connection with 
outpatient care. 

• For the BHO, CCSI has provided a series of analyses (SPSS) including weighted trending 
analyses and weighted aggregate performance analyses across the 33 providers in multiple 
counties. CCSI also developed, delivered, and provided training for a multi-dimensional 
Provider Profile Report Card (pictured next page) which portrayed outcomes, targets, and 
performance across time periods and measurement domains. This can be used to 
recommend CQI strategies to boost readiness for managed care. We also performed a 
predictive analysis (SPSS) around the use of services in relation to race/ethnicity, gender, 
and age. 
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• For the HH, which covers 22 Upstate NY counties, we 
developed the start-up data structure which involved a series 
of Excel and Access tools over a SQL data platform to make 
raw Medicaid data from NYS actionable at the HH governance 


P- level. We also performed extensive analyses (SPSS) across an 
■ integrated dataset (drawing from 4 separate datasets) to 
H develop initial quality assurance results for individual HH 
providers. Through this format, we were able to link 
individuals across the individual stages of engagement. 


• Currently working with the HH to help develop quality assurance and improvement 

mechanisms for the newly deployed NetSmart care manager software. One aspect of this 
development is the capability to use the data to identify individuals not receiving adequate 
care based on their acuity level. 

2. Monroe County Deptartments Of Human Services and Public Health, Healthy Futures Grant : 

Categories: Conducting Research, Translating Research to Practice. 

CCSI worked with the Monroe County Departments of Human Services and Public 
Health to design and administer the evaluation components of a three-year Center for Disease 
Control translation grant called "Healthy Futures." In close collaboration with the principal 
investigator and the contracted provider agencies (private and public), CCSI developed a 
comprehensive range of qualitative and quantitative measures to assess the impact of Healthy 
Futures on local child welfare practices. CCSI's team observed and analyzed the 
implementation of evidence-based practices in the areas of supervised parental visitation, 
trauma-sensitive therapies, parent skill-building, and primary pediatric care for youth in foster 
care. This translation evaluation focused on the fidelity of clinical practice, the extent of 
practice implementation within individual agencies, the numbers of youth and families served 
by the grant, and the impact of the grant on organizational behavior and provided local 
authorities with vital information about the feasibility and effectiveness of multi-agency 
systemic improvement efforts. CCSI's consultants have utilized strategies such as stakeholder 
interviews, focus groups, validated assessments of individual and organizational factors, and 
custom-designed evaluation tools. 

3. Monroe County Office of Mental Health's Recovery Connection Program: Category: 
Conducting Research 

The Recovery Connection program was developed to address the needs of individuals 
who frequently receive inpatient detoxification or rehabilitation services and who have 
experienced difficulty engaging in sustained addiction treatment. The need for this program 
was identified through feedback from local provider agencies and supported through the 
analysis of Medicaid claims and other data sources. CCSI collected program data using several 
methods and outcomes or findings were then used to support decisions for both policy and 
practice, such as which clients were being underserved in the behavioral healthcare system, the 
areas of deficit, and potential solutions. 
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New York University (NYU) - McSilver Institute for Poverty, Policy, and Research 

1. Center for Collaborative Inner-City Child Mental Health Services Research (CCCR) : 

Categories: Conducting Research, Translating Research to Practice. CCCR was established in 
collaboration with: 1) the NYS Office of Mental Health, and 2) two family and community 
advocacy boards representing inner-city youth, families, and communities (Bronx Community 
Collaborative Board and the Family Advocates Research Board (FAR Board). The aim of the 
CCCR is to organize multi-disciplinary researchers on the conduct of child mental health services 
research to improve the overall health and mental health of vulnerable youth. The CCCR 
specifically focuses on the development and testing of novel clinical practices and service 
delivery models that are informed by existing empirical findings and the outcomes associated 
with intensive collaboration between researchers, practitioners, youth, and families living in 
inner-city neighborhoods. 

2. STEP-UP: Promoting Youth Educational Success, Future Options, Mental Health and Positive 
Development within Inner-city High Schools & Local Communities : Category: Translating 
Research into Practice. 

NYU-McSilver and partners have developed STEP-UP, a program designed by African- 
American and Latino adolescents (14 to 17 years of age) experiencing significant academic, 
social, and emotional issues for teens in similar circumstances. In partnership with two urban 
high schools, STEP-UP was designed to connect young people with mental health clinical 
supports, positive young adult role models, supported internships, employment opportunities 
and non-stigmatizing family-level interventions. The ultimate goal of STEP-UP is to offer 
intensive programming for young people within their natural contexts (home, school, and 
community) in order to organize supports geared towards high school graduation, college 
preparedness, and eventual employment success. 

3. The Center for Mental Health Implementation and Dissemination Science in States for 
Children, Adolescents, and Families (IDEAS) : Categories: Translating Research into Practice, 
Predictive Modeling. 

IDEAS addresses challenges associated with closing the gap between research and 
practice in state systems serving children and families. The overarching aim is to improve 
implementation of evidence based practices (EBPs) through rigorous testing of strategies that 
target family and agency contexts. The Center's research studies use experimental methods 
and mixed qualitative and quantitative approaches to examine organizational, contextual, and 
family support approaches to improve client and system outcomes. 

4. Clinical Technical Assistance Center (CTAC) : Category: Translating Research into Practice. 

NYU-McSilver Institute, in partnership with the NYU Child Study Center, houses the 
Clinical Technical Assistance Center (CTAC), which is a training, consultation, and educational 
resource center available to the nearly five hundred mental health clinics in NYS. The purpose 
of CTAC is to advance the effective and efficient provision of clinic treatment to adults, children, 
and families who rely on public sector services to meet their mental health needs in three key 
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areas: 1) best organizational practices; 2) evidence-informed engagement and; 3) evidence- 
informed, family-centered service delivery. 

CTAC offers a variety of trainings and resources to help improve clinic operations and 
services, including webinars, seminars, and program trainings. NYU-McSilver employs effective 
approaches that reflect the context of day-to-day clinic realities. Specifically, CTAC's 
coordination of the Business Effectiveness Assessment Module (BEAM) and Business 
Efficiencies and Effectiveness Project (BEEP) exemplifies the type of projects that McSilver will 
draw from to support this proposal. BEAM is a collaborative of seventy enrolled clinics across 
NYS that NYU-McSilver guided through comprehensive self-assessments and planning processes 
to use data to identify clinical and operational outcome measures. BEEP was designed to assist 
participating clinics assess and identify opportunities to redesign financial structures and 
practice process flows, and to establish best practices to share across communities and 
agencies. These two initiatives have resulted in increased financial and operational health (and, 
thus, viability) to clinics, expanding their internal capacity and fostering participation with other 
organizations. 

2) Describe relevant experience in working with human services practitioners (including 
government analysts, clinicians, social workers, leaders). 

Coordinated Care Services, Inc. 

CCSI has a long history of working with practitioners in the health and human services arenas 
throughout New York State. Examples already noted above include: 

• New York Care Coordination Program: NYCCP, BHO, HH. 

• Monroe County Depts. Of Human Services and Public Health: Healthy Futures 
Additional examples are as follows: 

1. Monroe County of Mental Health : CCSI has provided a broad array of behavioral health 
management services to the Monroe County Office of Mental Health (MCOMH) since 1993. 
Current areas of support include: 

• Contract and Performance Management: CCSI manages behavioral health and 
developmental disability service contracts with 21 mental health service providers, 16 
chemical dependency providers and 7 developmental disability agencies within Monroe 
County. To respond to the increasing emphasis on accountability at the State and local 
levels, CCSI and MCOMH staff has worked to develop systems and practices to collect and 
analyze the performance data needed to support ongoing monitoring activities. 

• Quality, Accountability, and Data Management: CCSI provides the staffing and resources 
necessary to measure the effectiveness and performance of behavioral health services and 
maintains access to the information systems needed to support planning, policy and 
performance oversight activities in Monroe County. Work in this area involves using 
multiple data systems to measure performance at both the program and systems levels. 

This has included obtaining Medicaid claims and encounter data from the NYS Department 
of Health and analyzing these data to better describe care patterns and identify service 
gaps. More recently, staff has been trained to use a data analysis system developed by 
Salient Technology to support this work. We are proficient in using SAS, Stata, and SPSS. 

• Policy and Planning: Together with the MCOMH Chief for Policy and Planning, CCSI staff has 
worked to analyze relevant State and national policies, including the local implications of 
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these policies, conduct community needs assessment activities, facilitate processes to 
identify priorities, and develop strategies to address them. 

While CCSI focuses resources in MCOMH, CCSI does similar work in 28 counties in Upstate NY 
and thus is accustomed to county operations. 

2. Rochester/Monroe County Homeless Continuum of Care : CCSI was recently selected as the 
Collaborative Applicant and fiduciary agent for the Homeless Continuum of Care (CoC) Project 
in this jurisdiction. As the Collaborative Applicant, CCSI works with leadership in the homeless 
system - government, provider agencies, and housing developers - to support the community 
in developing and implementing plans and strategies to address homelessness. Using the 
Homeless Management Information System linked with other relevant data sets (e.g., 

Medicaid, behavioral health) CCSI works with CoC leadership to support data-informed decision 
making for planning and HUD funding allocations. 


3. Conference of Local Mental Hygiene Directors (CLMHD) : CCSI provides an array of technical 
assistance support to the CLMHD, a statewide group of the Directors for Mental Hygiene 
Services for all counties in NYS. As part of this technical assistance, CCSI works with the 
leadership from multiple NYS Agencies and Counties to develop and implement a Data 
Dashboard to support systems planning and development at the County and Regional levels. 
The process to develop the key indicators for the dashboard display is on-going, with indicators 
being added as key elements are identified, vetted by leadership (are we using the best 
available indicator to address the question we want to answer?) and populated into the 

— ■■ ■ - — dashboard upon conclusion of the process. The 

dashboard is intended to support County 
leadership in decision making, using available 
non-PHI data sources and providing trended 
views via business intelligence tools (Tableau). 
Data can be easily exported to excel formats for 
manipulation by the user. Current areas of the 
dashboard covering the behavioral health and 
developmental disabilities populations include 
Prevalence, Service Utilization (Medicaid), Hospital Admission/Readmission, Continuity of Care 
(Post-Hospital Discharge), and Homeless Services (Point in Time Data). CCSI has also developed 
an accompanying Fiscal Dashboard with cost information to support the Counties. Several 
other areas of the dashboard are in development for inclusion in the near future (e.g.. 
Jail/criminal justice. Child Welfare/System of Care). 



Criminal iustka Syttam Intarfaca 


4. Family Assessment Response (FAR) is an alternative response to families who have been 
referred to the Child Protective Services (CPS) system and who meet certain eligibility criteria. 

A solution-focused, strengths-based, and family-led approach is at the core of FAR, with the 
additional goals of identifying the need for, and then linking families to services. The traditional 
determination (i.e., indicated or unfounded) is not made on FAR reports. Implementation of 
FAR began within the Monroe County Department of Human Services (MCDHS), Child and 
Family Services Division, in January of 2010, with CCSI evaluating its implementation. The scope 
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of CCSI's evaluation work covered not only the process of implementing FAR (such as family and 
caseworker perspectives on the FAR approach), but also the impact and cost. Methodological 
techniques such as family questionnaires, child and family systems data, staff questionnaires, 
focus groups with staff, case-specific questionnaires, and financial analyses are being utilized. 
Specific data being collected include family satisfaction, stress levels, emotional responses to 
FAR, case practices and components, and family characteristics. Federal outcomes such as 
additional CPS reports or placement of children in foster care, staff perceptions of FAR impact, 
types of services for families, and service cost difference between FAR and non-FAR are also 
collected and will serve to provide an in-depth analysis of the FAR implementation process and 
goals achieved. Given that both FAR and CPS caseworkers sit on the teams involved in the FAR 
implementation, CCSI has been offered a unique opportunity for a rigorous and meaningful 
comparison between the "traditional" and "experimental" approach. Further, family 
perspectives data are being collected when the family enters CPS or FAR, at closing and then 3- 
months post-closing to assess any changes in families' experiences over time. 

New York University (NYU) - McSilver Institute for Poverty, Policy, and Research 

NYU-McSilver carries out extensive work in the community. Examples already noted above 
include: 

• Center for Collaborative Inner-City Child Mental Health Services Research (CCCR) 

• Clinical Technical Assistance Center (CTAC) 

An additional example is as follows: 

1. Community Partnerships to Prevent Urban Health Risks (CHAMPIONS) : The primary aim of 
CHAMPIONS is to advance scientific knowledge about research/community partnerships that 
can serve as the foundation for local adaptation, delivery, and testing of evidence-based 
prevention programs for inner-city African American and Latino adolescents. The project 
examines factors that influence urban, low income parents' willingness to collaborate in the 
delivery of evidence-based programs designed to reduce adolescent pregnancy, STDs, and HIV 
infection among inner-city minority youth. In addition, the study tests the impact of a 
community mentorship and training component on implementation, as well as on youth sexual 
risk intentions and behavioral outcomes. 

3) Describe any collaborative projects on which your organization worked, naming partner 
organizations. 

Coordinated Care Services, Inc.: 

CCSI has a long history of supporting and working as part of multi-stakeholder 
collaboratives. Some recent examples include: 

1. The Clinic Technical Assistance Center (CTAC) : As noted above, CTAC is funded by the NYS 
Office of Mental Health to provide training, consultation, and educational resources to all 
licensed clinics in NYS. The goal of CTAC is to provide clinics with a set of technical assistance 
and training tools that promote effective care through efficient practices while also addressing 
the challenges associated with the recent changes in clinic regulations, financing, and overall 
healthcare reforms. As a part of the Center's leadership, CCSI collaborates with a strong set of 
New York state-based partners, including: New York University (NYU) Silver School of Social 
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Work, McSilver Institute for Poverty, Policy and Research, Institute for Community Living, Inc., 
Families Together, and New York Association of Psychiatric Rehabilitation Services, Inc. 
(NYAPRS), to provide training and technical assistance related to the implementation of 
evidence-based practices, billing practices and fiscal management, the use of data for quality 
improvement and decision making, and treatment supply and demand. 

2. New York Care Coordination Program : As noted above, CCSI has supported this multi¬ 
county, multi-stakeholder collaborative aimed at improving services for individuals with serious 
mental illness. Members include: County Departments of Mental Health (Cayuga, Chautauqua, 
Wayne, Onondaga, Rensselaer, Westchester, Monroe, Erie and Chemung); Provider Agencies 
(Hillside Family of Agencies, Onondaga Case Management Services, Inc., Lake Shore Behavioral 
Health, Huther Doyle and Strong Behavioral Health) Peer and Family Representatives (Peer 
Advocate, Community Based Peer Initiatives Transitional Living Services, Mental Health 
Association of Olean, Housing Options Made Easy, Recovery Services, Center for Community 
Alternatives). 

3. NYS Success: Connecting Systems of Care with Children and Families : In 2012, SAMHSA 
awarded Upstate New York with a four year, four million dollar grant to support broad-scale 
operation, expansion, and integration of systems of care (SOC) through the creation of a 
sustainable infrastructure. CCSI provides overall project management for this cross-systems 
initiative. Collaborators and partners include: Families Together in New York State, the 
Conference of Local Mental Hygiene Directors, YOUTH POWER!, and the Ad Council. 

With the assistance of a designated core implementation and planning team, all 55 
upstate NY counties are expected to successfully integrate and sustain the SOC values, 
principles, and practices within their own communities. Over the course of four years, each 
county will be brought into the NYS Success network as a member of one of four learning 
collaborative cohorts. 

4. District of Columbia Depts. Of Behavioral Health and Child & Family Services : CCSI is 
currently partnering with Community Connections of New York and State University at Buffalo 
School of Social Work as the lead evaluator on several projects with the District of Columbia 
aimed at strengthening services for children and families. Two are described below: 

• A system of care grant, funded by SAMHSA, is a multi-year initiative aimed at supporting 
the development and expansion of sustainable structures that provide children and 
families with access to the full range of needed community-based services. The 
District's priorities include expansion of evidence-based practices, increased informal 
parent-to-parent and youth-to-youth support, community education to decrease mental 
health stigma, support early identification and access to services, and development of a 
reinvestment strategy to sustain services. 

• Trauma-informed child welfare practice is an initiative funded by the DC Administration 
on Child, Youth, and Families. The intent is to facilitate transformation of the District's 
current child welfare programs into processes which are trauma-informed, contributing 
to measureable improvements in social and emotional well-being of children in foster 
care. 
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New York University (NYU) - McSilver Institute for Poverty, Policy, and Research 

NYU-McSilver works with a number of partners for a variety of purposes. Examples already 
noted above include: 

• Center for Collaborative Inner-City Child Mental Health Services Research (CCCR) 

• Community Partnerships to Prevent Urban Health Risks (CHAMPIONS) 

• STEP-UP: Promoting Youth Educational Success, Future Options, Mental Health and 

Positive Development within Inner-city High Schools & Local Communities 

• Clinical Technical Assistance Center (CTAC) 

Additional examples are as follows: 

1. YMCA, National Alliance on Mental Illness, Brooklyn Academy of Music : NYU-McSilver has 
extensive experience providing evaluation services for large and varied organizations such as 
the YMCA of Greater New York, the National Alliance for Mental Illness (NAMI), and the 
Brooklyn Academy of Music (BAM). Since January of 2013, NYU-McSilver staff has contracted 
with the YMCA to provide program evaluation services for three major programs aimed at 
identifying sub-groups of individuals at high risk for crossing over into the criminal justice or 
other systems and need for preventive services using client information and outcomes data. 

For BAM, NYU-McSilver works with its Education department to assess the impact of a social 
justice program on youth program participants' academic and psychosocial outcomes. This 
evaluation is designed to predict the potential impact of the program on the community and 
identify new measures and data collection needs to help increase effectiveness. Evaluation 
services provided to NAMI include development of an evaluation protocol, survey tools, and 
staff training for three programs - Basics, Family to Family, and Peer to Peer -to measure the 
impact of services on participant knowledge of managing their mental and physical health. 

2. Community Collaborative Board (CCB) : The Bronx Community Collaborative Board is a 
formalized partnership between representatives of urban neighborhoods and researchers. 
Currently, the Collaborative Board oversees a number of federally and locally funded research 
studies focused on designing, delivering, and testing family-based prevention and intervention 
services for elementary, junior high and high school age urban youth and their families. The 
Collaborative Board consists of 30 members, urban youth, parents, school staff, representatives 
from community-based agencies and researchers. 

4) Describe the way your organization or collaboration would manage the process of working 
with DHS to design and implement decision support tools and predictive analyses. 

We are seeking to develop the foundation of rigorous discovery necessary to advise the 
development of decision support strategies for 2 questions posed in the RFP: A) "How can we 
assist workers to use the data collected on client (family) risks, strengths, and needs to target 
interventions (including specific evidence-based interventions)?" and B) "Which clients 
receiving behavioral services may not be receiving services at the appropriate level of care?" 
Before we can begin to do begin any work, we must develop be prepared to do so in line with 
the current processes within that environment. It is important that this project is conducted in 
close alignment with the culture, goals, and values of the AC DHS. The AC DHS Vision 
Statement seeks 
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"To create an accessible, culturally competent, integrated, and comprehensive human services system 
that ensures individually tailored, seamless and holistic services to Allegheny County residents, in 
particular, the county's vulnerable populations." 

We believe that the key operational word is integrated. AC DHS has as its core mission to seek 
integration across its several client-facing Offices (Intellectual Disability and Behavioral Health, 
Community Services, Area Agency on Aging, Children Youth and Families) to create a single, 
holistic entity. The central component of integration is the Executive Deputy for Integrated 
Program Services. We expect to work closely with Patricia Valentine across the execution of 
this project, which we see as being very easily shaped to meet specific needs, services, and 
directions to promote programmatic integration. There is a stated desire in AC DHS to increase 
the focus on continuous quality improvement and to bring Predictive Analytics into the service 
delivery equation to inform better practices and procedures. In the same vein as the initiatives 
to streamline program monitoring across the different oversight agencies to coordinated 
planning and policy development, it is important to AC DHS to have this project conducted in 
accordance with the focus on integration. 

Setting the Course: The mechanism to achieve this is to work closely throughout the project 
with AC DHS senior leadership. This will be evident from our kick-off meeting going forward. 

We propose to first work with administrative/executive staff (at least one 1.5 hour planning 
meeting) to develop an overall CQI strategy around decision support tool development and 
predictive modeling. We will also seek a better understanding of the AC DHS operations 
environment and culture including focus, approach, timeline, components, and key 
participants. We will also work to identify key areas of importance for AC DHS to establish a 
starting point for this work to begin. These initial meetings will involve the following 
components: 

• Discussion and development of concordance of CQI philosophy and approach within the 
culture of the AC DHS. While discussion will be on the development of CQI planning for 
decision support and tool-building, it will be imperative for Project staff (CCSI, NYU- 
McSilver) to understand the culture, focus, direction, and values of AC DHS. 

• Developing an overall vision for the role, place, and scope of decision support within AC 
DHS. 

o Qverall strategy to identify key areas requiring decision support 

• Discussion and description of the overall process of developing predictive models which 
can lead to decision support. 

o Qutcome criteria, predictors, developing the model, describing analysis and 
output, and deciding how to address bringing results to become decision support 
tool(s). 

• Determining the type and culture of roll-out that will most likely result in success 
(making a game plan). 

o Underscore the need to identify one AC DHS contact person (and one back-up) 
with whom the project can work to properly design and implement the process 

• Helping us gain a greater understanding of the data system/warehouse 

o Review of AC DHS data warehouse design and elements 

o Assessing need for new data exchange agreements or expanding existing 
agreements 
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o Identify what data are there, what data are missing, what additional data might 
be needed 

5) Describe experience or approach to working with an existing IT vendor to 
implement/integrate solutions. 

Both CCSI and NYU-McSilver have significant experience in working with IT vendors to 
develop, deliver, and support solutions that improve services and practice. 

Coordinated Care Services, Inc. 

Since 2006, CCSI has partnered with Innovative Solutions, a Rochester-based IT firm that 
provides a full array of technology solutions and professional services to many businesses and 
organizations within Upstate New York, with a specific focus on the health care and not-for- 
profit sectors. Key service areas include website development and support, custom software 
development, network engineering and design, and system integration. As a key technology 
partner. Innovative has provided IT support for a number of priority initiatives. CCSI has 
worked with Innovative Solutions and other vendors around a number of software builds, two 
of which (PCMS and ARES) are presented below. A third was with ICF International (OBP). 

• Performance and Contract Monitoring System (PCMS): Developed a (SQL-based) 

contract and performance management solution using a 
combination of existing software and customized 
development. This software is used across NYS counties 
to track performance deliverables in relation to targets 
and baselines. This is used for monitoring progress of 
contracts as well as data tracking for quality 


Addiction Recovery Empioyabiiity System (ARES): CCSI designed the web-based ARES 
system (SQL-based) to support the Monroe County Department of Human Services in 
the monitoring of Temporary Assistance clients referred to chemical dependency 

treatment services. Supported by CCSI for nearly a 
decade, ARES has been extremely helpful to the 
County in performing this oversight role effectively 
and efficiently. In addition to collecting monthly 
attendance reports from participating providers, 
ARES collects quarterly Progress Reports for clients 
attending outpatient treatment. This automated 
system allows employability decisions to be made 
more efficiently, thus helping clients to move more 
quickly toward self-sufficiency. As Monroe County DHS moved to a task-based system 
of processing cases, ARES was enhanced to distill task notifications down to essential 
actionable tasks which alert county staff to take case actions based on poor attendance, 
discharges from treatment and determinations of ability to work. This efficient, real¬ 
time exchange of pertinent treatment information between ARES and DHS has 
contributed to a high number of client's successfully completing Chemical Dependency 
treatment, moving on to their next stages of self-sufficiency. 
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Outcomes-Bosed Prescribing (OBP): CCSI is a partner working on the next stage of 
development of an award-winning innovative OBP software prototype developed by the 
PI for this proposal (Crilly). General information about how medications work and their 
side effects can be readily accessed by clinicians and patients through abundant online 
resources. While access to these resources is important for patients, the applicability of 
general medical information to the individual's own experience remains limited. This 
gap impedes the process of better engaging patients in decisions for their own 

healthcare. Using a concept of 
Individualized Comparative 
Effectiveness Research, we 
developed a working prototype of a 
practical tool which draws upon 
enterprise electronic health record 
data to help patients and clinicians 
better predict individual outcomes 
when making decisions for new or 
additional treatments. The 
development of this prototype 
software used data drawn from the 
VistA electronic health record 
system of the US Department of 
Veterans Affairs. Dr. Crilly and team 
worked in close collaboration with software development specialists ICF International to 
complete the prototype build. 

New York University (NYU) - McSilver Institute for Poverty, Policy, and Research 

• NYU-McSilver was recently awarded a $699,735 grant from the National Institute of 
Mental Health to collaboratively create and pilot web- and app-based homework 
support within child-serving agencies across New York City. Heading the development is 
Co-Investigator for this proposal Dr. Cleek along with collaborators Dr. Kimberly 
Hoagwood of the NYU School of Medicine, Dr. Anil Chacko from CUNY Queens College, 
Kimberly Johnson from the University of Wisconsin, Madison, and Nikolaos Kazantzis of 
the La Trobe University in Melbourne, Australia. 

• NYU-McSilver has also been awarded a grant for over $300K from the NYS Health 
Foundation to help mental health clinics throughout the state integrate outcome 
measures into their clinical treatment. The use of outcome measures helps inform 
treatment decisions, track goals, and monitor client outcomes over time, which will be 
vital as changes to the NYS Medicaid regulations will require providers to show 
improvements in client outcomes in order to be reimbursed for providing treatment 
services. 

• NYU-McSilver is also creating a searchable database of outcome and performance 
measurement tools that mental health clinics can use to identify and access 
standardized assessments based on client characteristics and symptoms. The use of this 
database is intended to help clinics assess the effectiveness of treatment and for clinics 
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to be able to demonstrate to funders the impact of the services they deliver. By 
tracking clinical outcomes in a systematic way, clinics will be better able to determine 
the impact of different interventions and make data-driven treatment decisions. 

• NYU-McSilver is developing an online portal that clinics can use to enter data, measure 
their progress, and benchmark themselves in a de-identified way to other clinics around 
in NYS. These data will allow CTAC to identify and disseminate best practices based on 
client characteristics and diagnoses. 

• NYU-McSilver has been involved in a number of other technology-related projects, 
including as a subcontractor for the Health and Hospitals Corporation ( NYC) to connect 
7 community-based agencies to a NYC Regional Health Information Organization (RHIO) 
either through a direct connection or a portal. They also managed a HEAL-17 project for 
a community agency and have been involved in a process of discovery, selection, and 
implementation of electronic health records for a community based health home. 

B. Project Description: 

1) Identify project goals and objectives. 

Frontline staff, managers, and executives within the AC DHS are faced with decisions 
daily. Some can be made based on experience and others by readily available report data. 
However, these options do not work for everyone, nor do they work under all conditions and 
for all situations. They especially do not work for the most difficult and serious decisions often 
required of DHS workers during the course of their jobs, such as: 

• Should the child remain in the home? 

• When should a child be returned from out-of-home care? 

• Which clients receiving behavioral services may not be receiving services at the 
appropriate level of care? 

Decisions are often made without having all of the necessary information (despite its being 
available with the agency's data system), perhaps impacted by fatigue, influenced by bias, and 
required to do so at a fast pace. Yet, agencies and frontline workers want to do the best job 
possible and are constantly searching for ways to improve. One area for improvement which 
can have the biggest potential impact on quality and safety across DHS programs is access to 
the right decision support tools at critical points in the intervention continuum. Key process 
steps such as clinical judgment and risk assessment can be aided by having access to key pieces 
of data (e.g., that a particular family has had more than 3 child protective calls this week, that 
one parent is in jail, and that SSI has been discontinued; that a child may be the past target of 
abuse by caregivers, has been arrested twice recently, and is on a series of potentially counter- 
indicated medications from health and mental healthcare providers). 

It is important to note that AC DHS already has this level of data use which is itself a 
form of decision support. A recent report (GAO, 2013) showcased the usefulness of the AC DHS 
data warehouse in that it has increased the amount of client-level information intake workers 
have at their disposal. Several examples are given about how these data have helped support 
decision making and other important tasks across DHS (p. 40): 

• Data can help identify whether reported person or another family member receives other government 

services (e.g., behavioral health) or has a criminal history 

• Access to demographic information collected by other departments, (e.g., state Dept, of Public Welfare) 
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• Speeds up ability to get information on a family when limited or inaccurate information is provided at intake 

• Adds material for probing in their questioning with a client who may not always be forthcoming. 

• Helps manage time, such as by knowing who else is living in the house and who they need to interview before 
conducting a home visit. 

• Ensure their own safety because they have better background information on household members before 
making the home visit (e.g., information on criminal history or mental health status of household members). 

This decision support is used informally in that users define their own needs and it is by their 
own initiative that this is used. It is natural that this occurs but can make the design, use, and 
delivery of the decision support if offers inconsistent and haphazardly distributed. 

This raises a series of complex issues around the conceptualization of the development 
of decision support tools. While the existence of data makes the potential of creating decision 
support tools possible, inadequate conceptualization of the utility of those data can actually 
limit its scope. With access to large amounts of data, it can be tempting for inexperienced 
developers to equate sheer volume of data with adequacy of content breadth. Doing so raises 
the risk of unintentionally narrowing or reducing the scope of discovery where the complex 
issues facing AC DHS could be thought to be well-addressed within those existing data alone. 

We will bring a more contextual view, that it is important to have a multiple-levels-of-analysis 
perspective when seeking solutions to very complex problems (Cicchetti and Blender, 2006). 
When we reach the point of considering solutions (including building new software) to bring 
data to the individuals who need them, it is the development of the decision support elements 
and the appropriate blend of involving staff and non-staff experts, research, and consumer 
input which promote multiple levels of analysis. This process of convening and evaluating 
current practice, research, organizational goals and cultures, and many other non-data, non¬ 
technical detail are critical to the proper design, implementation, and use of such tools (Foster 
and Stiffman, 2009). This proposal seeks to ensure that these elements are all properly in place 
and serves to guide the development of the eventual decision-making technology. 

As we submit this proposal, we are well aware that the AC DHS is highly qualified and 
has many resources, but realize that those resources are already fully deployed in its day-to-day 
operations. As mentioned in the RFP, software builders can build software adequately but 
most do not have the expertise around DHS functions, process, and purpose. It is difficult and 
rare to do both very well. Although our group is experienced in both areas, we will focus on the 
important pre-build preparation and leave the software building to others. Indeed, without this 
piece, developers and DHS workers may not easily find the common ground upon which to 
exchange ideas and advisement. Keeping with the overall theme of integration across DHS 
departments, the goal of this proposal is to provide an iterative model of continuous quality 
improvement around decision support by building the capacity to develop the key components 
of decision support to the point of the software build. A key consideration for AC DHS will be 
whether a particular decision support solution to a particular problem actually requires an 
expensive software build, to ensure that the funding of the actual decision support tools 
should only be considered at the point at which it is determined that the approach or 
intervention currently in place, around which the decision support will be built, is optimal, 
clearly defined, and well understood. 

Basing decisions on inadequate models (i.e., inefficient or insufficient processes or 
interventions) cannot provide good value to an end product, so the discussion of creating 
decision support tools should best occur following a reasonable process to determine the 
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adequacy of the current approach, adjusting it as necessary, and then, after evaluation, going 
forward with building decision support. Also impacting the effectiveness of a tool is the 
expected end result, the question of what steps should then be taken by the AC DHS worker. 

To get such an approach correct can take considerable time and resources. It is also important 
to maximize the impact of this process, which is a classic model of CQI-building. Weaving this 
work into a strategy of department-wide (but very focused) CQI it could produce synergistic, 
sustainable effects beyond the scope of this project. The Goal of this proposal is that through a 
rigorous process of ensuring clear definition of the scope of each question and the input of 
experts, sufficient predictor and outcome variables will be identified to build efficient predictive 
analytic models which will advise the development of highly relevant, high quality decision 
support tools . While the processes we propose here can be used to explore any of the 
questions listed in the RFP, we intend to focus on 2 areas identified in the RFP: A) "How can we 
assist workers to use the data collected on client (family) risks, strengths, and needs to target 
interventions (including specific evidence-based interventions)?" and B) "Which clients 
receiving behavioral services may not be receiving services at the appropriate level of care?" 
The goal of this project will be achieved within 4 objectives, activated in 2 stages. 

Stage 1: Foundation Determination: 

This first stage will have 2 purposes: 1) to develop methods to apply discovery and rigor 
to the determination of areas requiring decision support, and 2) to develop methods to 
determine whether the current practice/AC DHS intervention is sufficient to serve as a basis for 
predictive modeling. 

Objective 1 : To work collaboratively with AC DHS to identify key areas requiring decision 
support and to convene staff to define their scope with the purpose of developing specific 
outcomes criteria around which to build predictive analytic models. 

Objective 2 : To identify and quantify potential predictor variables for use in predictive analytic 
models through content expertise and knowledge and review of the literature. This may (if 
necessary) also include the option of revising approaches in critical areas lacking in sufficient 
rigor so as to move to Stage 2. 

Stage 2: Predictive Analytic Model Construction: 

Proceeding to Stage 2 will only occur after Stage 1 objectives have been satisfied. Stage 
2 focuses exclusively on Model development for only those items shown to have sufficient 
internal rigor to be amenable to or able to make use of decision support tools. 

Objective 3 : To pull together the components created in Objectives 1 and 2 to formulate, test, 
and analyze predictive models and construct the basis for creating decision support tools. 
Objective 4 : To develop specific responses to the outcomes of the decision support. These will 
include processes to develop new interventions designed to achieve the desired result and 
recommendations around the design or development of appropriate decision support tools, 
which could range from simple delivered data feedback via standard business intelligence tools 
like Tableau or Jasper to software builds. We will then participate in and advise the product 
testing, product revisions, roll-out, and evaluation. 

2) Describe in detail the services/consultation to be provided: 
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Objective 1: To work collaboratively with AC DHS to identify key areas requiring decision 
support and to convene staff to define their scope with the purpose of developing specific 
outcomes criteria around which to build predictive analytic models. 

CCSI and the NYU-McSilver Institute will work collaboratively across all 4 objectives but 
will each have varying degrees of involvement depending on the objective. Objective 1 will 
primarily be the responsibility of CCSI which will convene AC DHS staff and service consumers 
to build this process. We will approach this from the overall philosophy of instilling continuous 
quality improvement (CQI) processes for the purpose of identifying areas for which decision 
support tools would add value to both DHS staff and clients. This is in keeping with the long¬ 
standing collaborative approach AC DHS has used to involve stakeholders since its inception 
(e.g., the formation by OBH of the Change Management Team in 2007 and a workgroup for using social media to 
combat stigma - Allegheny County Mental Health Plan, 2012: pp. 10 and 15 respectively). 

As outlined above, we will first work with administrative/executive staff (at least one 1.5 
hour planning meeting) to develop an overall CQI strategy and gain an understanding of the AC 
DHS environment including focus, approach, timeline, components, and key participants. We 
will also work with them to identify key areas of importance for AC DHS to establish a starting 
point for this work to begin. 

Primary Task: Setting Predictive Analytic Model Parameters: AC DHS Staff and Consumers 
Focus Group(s): On the basis of the information gathered above. The Project team will develop 
well-placed focus groups aimed toward the critical areas identified within the administrative 
sessions. We will work with the identified AC DHS project contact person(s) to create the 
groups and to include a sufficiently broad spectrum of stakeholders (administrators, managers, 
line workers, consumers). We will hold a series of 2-5 separate focus group meetings around 
the identified issue. We anticipate that the first series of processes will require more 
involvement as we start this process but will become less necessary over time. Our intent will 
be to eventually "hand off" this discovery process to the AC DHS team as part of their overall 
decision support CQI process. During this start-up phase, the Project team will coordinate and 
facilitate this process. The objective of the focus groups will be to form the basis for the 
predictive modeling. The work of these groups will be to: 

1) Specifically identify and clearly define the intended client target population, group, or 
issue of a decision support mechanism. It will be this identification which will define the 
scope for the predictive modeling and thus advise the direction of decision support tool. 

2) Identify and clearly define specific outcomes (i.e., dependent variables) expected or 
desired from this process. This will set the target for the predictive modeling. 

3) Determine a framework for identifying potential predictor (i.e., independent) 
variables for the predictive modeling based on AC DHS data, DHS best practices, 
practitioner experience, and consumer experience. This will also include the 
determination of the sample size of the target group/issue as a parameter for the next 
steps. This framework formulates the starting point for the work of Qbjective 2. 

4) Establish concordance between NYU-McSilver and CCSI on these components so that 
the NYU-McSilver team can immediately launch the next stage of work. 

Objective 2: To identify and quantify potential predictor variables for use in predictive analytic 
models through content expertise and knowledge and review of the literature. 
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Determining predictive analytic model components will fall under the purview of NYU- 
McSilver Institute. CCSI will participate by facilitating the crosswalk between the qualitative 
data from the focus groups to NYU-McSilver which will lead the development and refinement of 
the predictor variable set of the predictive analytic model(s). They will start with the input from 
the focus groups and will add the critical factor of providing a broader reach for informed input. 
The choice of predictor variables must be based on policies, best practices, clinical expertise, 
and continuous review and integration of literature findings and resources. Variables must be 
posed in ways that are clear and will result in productive and useful information. Their level of 
clarity will produce the best possible decisions. NYU-McSilver Institute has extensive 
knowledge expertise and is well-positioned to provide rapid turn-around of thorough literature 
reviews and after the infusion of expertise and knowledge of both adult and child populations 
typically involved with DHS. They have extensive experience working within DHS topic areas, 
particularly in the areas of poverty and child welfare. The Institute is closely aligned with the 
NYU School of Social Work and has access to a large group of faculty expertise and field 
placement agencies which could be tapped for additional perspective. 

NYU-McSilver will be involved as an observer in the focus groups outlined in Objective 1 
so will have good context for the task of identifying and qualifying predictive values. With the 
list of proposed variables as their guide, they will initiate the following: 

1) Assemble a small team of advisors (NYU-McSilver faculty, NYU Social Work Department 
faculty, field agency staff) and NYU-McSilver staff who will scope out the work process. 

2) NYU-McSilver staff will scour the practice, policy, and research literature to identify a field of 
potential predictor components. Identified components must be evidence-based or best- 
practice and will be compiled into a growing database of potential elements complete with 
reference, keywords, and particular potential prediction scope. 

3) NYU-McSilver staff will meet in frequently (e.g., twice-weekly) discussion sessions headed by 
the NYU-McSilver project lead and include select advisors to confirm variables to be kept or 
rejected until a consensus is reached that the field of potential variables is exhausted. The 
purpose of this will be to develop a database of vetted predictor variables that can be handed 
off to AC DHS to utilize over time in subsequent predictor analytics projects. Unlike a static list 
of potential variables collected and presented in a published paper (e.g., Begle et al., 2010), this 
will be a dynamic process which will require periodic updating, but will form an excellent 
foundation for developing predictive analytic models. 

4) When this process is complete, CCSI will bring the final list of proposed predictor variables 
back to the focus group to confirm that it is ready for the predictive analysis stage. 

Although Allegheny County has ongoing methodology training and competence 
oversight, one of the key components of a CQI process is the determination of areas needing 
improvement. Although unlikely, we may find that in the process of completing Objectives 1 & 

2 that the intervention or process currently in place does not have a basis in the literature, and 
therefore should not yet be the basis of a decision support initiative. This would present an 
opportunity for the organization to review high priority areas for improvement. 

Objective 3: To pull together the components created in Objectives 1 and 2 to formulate, test, 
and analyze predictive models and construct the basis for creating decision support tools. 
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Predictive modeling (PM) seeks to offer information to groups of "information 
consumers" to help them in planning and making decisions. PM can form the basis of a robust 
decision support system both for DHS workers on the front lines of care as well as for managers 
and administrators. In a broader context, decision support systems have become standard 
issue with many large electronic health records and are common in business to help manage 
investing, marketing, and risk. As such, output types can be quite variable. Some decision 
support features can be as simple as auto-alerts in a health record or more complex as helping 
DHS social workers estimate the success of specific children placed in specific residential or 
supervised setting. Properly understanding methods for their development and use and how to 
apply their output is critical to the entire process. We discuss this below and describe the 
choices of methods available to match to different needs and settings. 

Discussion of Predictive Modeling Methods 

There can be a tendency when approaching complex problems such as those seen in 
child welfare agencies to use methods that seem to make sense but are not fully adequate. For 
example, we may assume that the number of risk factors present accumulates to create a 
crushing load on an individual or group, thus explaining the existence and severity of a 
particular outcome - the concept of cumulative risk. While risk factors are important, assuming 
that the amount accumulation of these factors can therefore form a predictive model can be 
too simplistic for some applications. Similarly, predictive modeling is sometimes considered a 
form of logistic regression, where a series of independent variables are assessed for their 
influence on one or more outcome or dependent variables. While the interpretation of the 
value as a predictive value is often seen in the literature (e.g., Jarvenpaa et al., 2004; Rai et al., 
2006), this statistic represents "variance explained," a strength-of-fit measure which is a 
different construct than prediction. High R^ could indicate a strong relationship amongst data 
yet still have low predictive value (Barrett, 1974). Instead, predictive modeling methods 
include a series of steps to build the model. Parts of the model are predicated on statistical 
significance while others are not. For example, unlike the building of regression models which 
are generally built upon variables which have statistical significance, predictive modeling can 
gain information and value from variables that are not statistically significant (i.e., it is their 
significance - or not - which can have predictive value). These less experimental and more 
observational data can be more useful for prediction because they are freer to vary and 
therefore more closely resemble real-life situations and outcomes. 

Predictive modeling follows a number of steps. The first step is identifying the goal of 
the particular analysis. Predictive modeling is generally intended to promote the ability to 
predict outcomes for new sets of observations based on responses to past values (predictions if 
numeric outcomes, classifications if categorical outcomes). In AC DHS, the majority of data for 
use in predictive modeling already exist in the extensive data warehouse. Depending on the 
analytic question, outside data may also be required to add dimensions to the existing data. 
Once the goal is identified, the next step is to determine expected or desired outcomes 
(Objective 1). This is an important step because the more succinctly these can be defined, the 
more closely the data can be modeled to fit (or not). This step typically includes stratification of 
outcome types or dichotomization of several potential outcomes. Bi-modal predictions, that is, 
predicting which cases fall in the positive spectrum (excellent outcome, top performers) vs. 
those in the negative spectrum (very poor outcome, lowest performers) can be equally useful. 
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With our outcomes defined, we then switch to determining the best independent 
variables to help shape the model (Objective 2). This is best done by drawing on subject matter 
expertise, field experience, and full access to and ability to use a broad base of literature. This 
process is highly interactive and requires coordination, efficiency, and clarity of purpose to 
reach the level of usefulness that will lead to quality predictive modelling. Consideration of 
independent variables also depends on dataset characteristics. Generally, we need to ensure 
that there are sufficient cases to provide sufficient predictive power AND that we keep the 
number of predictive variables to a minimum to prevent "over-fitting" the model. In general, 
the number of outcome events should be at least ten times the number of variables in the 
model and the number of cases per variable should (generally) not be less than n=75. Missing 
data are also an important consideration in two ways: 1) we can eliminate missing values by 
imputing data, or 2) we can transform missing data into data itself - that data are missing from 
a specific variable may be important predictive information and added to the potential 
independent variable list. Understanding the database provides important perspective for 
determining variables. 

Once the outcomes and predictor variables are finalized, we then determine the sample 
group to study. AC DHS's large dataset is ideal for designing cohort comparisons. Ideally, we 
could base the predictive model on the entire group of individuals affected by a particular 
decision support element. In this case of what we presume to be AC DHS' sufficiently large 
population, we instead would use a split-sample design and create two randomly-assigned 
cohorts: the derivation (sometimes called "training" or "model") cohort and the validation 
(sometimes called "hold out") cohort. The derivation cohort is generally two-thirds of the 
larger group and it is the cohort upon which the model is developed and refined. The 
remaining one-third is the validation cohort, where the models' prediction abilities are 
estimated. 

In working with the validation cohort, the object is to find the best set of model 
components. We generally start out (using the derivation cohort) by determining at least one 
model for every condition. For example, if we had 2 outcome measures (A and B) and 3 levels 
of outcome (excellent, moderate, poor) we would identify at least 1 model for each of the 
combinations (A/Excellent, B/Excellent, A/Moderate, etc.) for a total of 6 models. Models 
should be able to include both linear and nonlinear variables and there are statistical methods 
to help control assumptions (e.g., restricted cubic splines can help avoid linearity assumptions) 
when using dichotomous and continuous data. At that point, it is important to determine the 
discrimination of the model - that is, its ability to discriminate between cases/events and non- 
cases/non-events. To achieve this, we use the receiver operator characteristic (ROC) curve, a 
standard measure of overall predictive discrimination. We look for a c-statistic (range of 0-1.0) 
at the very least above 0.50 but generally prefer closer to 0.75 (moderately good 
discrimination) or higher. Depending on the type and quality of both the independent and 
outcome variables, we may also perform calibration of the model, and use the standard 
Hosmer-Lemeshow statistic to determine a particular model's fit to the data. Such capabilities 
are standard in statistical packages such as Stata, SAS, R, and SPSS. 

Alternatively, we could develop the model on the entire group (no split-sample design) 
for the purpose of determining how well a model might predict in a new population sample. To 
do this, we would perform an internal validation procedure using bootstrapping techniques 
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(Efron and Gong, 1983). Briefly, bootstrapping allows us to generate multiple random samples 
from the same dataset and test how well (consistently) the model predicts the outcome. The 
bootstrap-corrected data are then ROC-analyzed. This approach is generally used for smaller 
samples in which using the split-sample design would decrease the available n and thereby limit 
the number of independent variables to be used and would likely contain insufficient outcome 
events in either group for prediction. 

Another form of predictive modeling involves artificial neural networks (ANN), most 
notably in child welfare by Zandi (2000) and Schwartz et al. (2004). Briefly, ANN is systems of 
computer-based learning capacity which, among other things, is able to identify patterns within 
sets of data. This is accomplished through a series of algorithms which infer functions from 
observational data. This is particularly useful in complex situations where the integration of all 
data "by hand" is very difficult yet is imperative in making a correct decision. Zandi (2000), 
Schwartz et al. (2004) and others have identified the decision making required by child welfare 
workers to be of this caliber of complexity. Schwartz et al. (2004) reported that ANN methods 
successfully categorized 89.6% of cases of child abuse in their study with small (0.6%, 1.9%) 
false positives and negatives. Zandi (2000) had similar results (90% classified) but with a higher 
false positive/negative rate. The challenge of ANN is the high level of expertise required to 
design, implement properly, and maintain the system. However, this type of "data mining" can 
have considerable application potential within a child welfare system, especially with the data 
capacities of the AC DHS data warehouse. 

Alternative/Emerging Predictive Modelling Technigues: 

There are other forms of predictive modeling which are emerging in the literature, 
catalyzed by the ready availability of large datasets of human experience data and the 
conceptualization of consumer self-determination and personalized medicine. Two are worth 
noting here, one of which was designed by the lead applicant. The first is from Alemi et al. 
(2009), who suggest new statistical methods to examine whether the treatment (or approach) 
to a particular consumer is likely to be beneficial. They recommend the use of the "k-nearest 
neighbor" statistic to determine the extent to which the recommended treatment was effective 
in consumers "similar" to the index individual. Going a step further, Crilly and Luu (in press, 
also included in Appendix) developed a model for utilizing data from a large electronic health 
record for patients similar to the individual index patient by diagnosis and on a number of 
descriptive (e.g., geographic location) and treatment criteria. These are important concepts 
which seek to minimize the likelihood of "bad" choices of care or intervention because they use 
a broad array of available information which can be applied to predicting the outcomes of a 
particular case. Having these different types of Predictive modeling methods in our toolkit can 
add value to the final decision support tool design. 

Which to Choose? 

This will be the key question AC DHS will face with this project. The process of predictive 
modeling will set the stage for choosing amongst a series of decision support tools and delivery 
methods. However, which tools, modeling approach, and delivery methods will work best? 

This will likely depend on the specific question and the intended target and use, so it is 
important to have a flexible and well-informed endpoint. To address this, CCSI and NYU- 
McSilver will again convene a group of users similar to the focus group structure outlined in 
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Objective 1 to examine the potential types of decision support solutions which would be 
appropriate (data dashboard, software build, off-the-shelf decision support software [e.g., 
TreeAge, Precision Tree], text message alert, information clip, or other mechanisms). 

Objective 4: To develop specific responses to the outcomes of the decision support solution. 
Regardless of the choice of decision support solution delivery, the intent is that it should 
influence a decision and, subsequently, outcome. Users of decision support need to 
understand what type of response or reaction is expected of them once the support has been 
used (e.g., go with recommendation, consult with colleagues, check with supervisor, or 
integrate into their own judgment). This, of course, is not the intent because human judgment 
backed by experience and knowledge of the components on the ground remain critical 
components. It is also a risk that workers will not use the tools. This can occur for any number 
of reasons, but the most likely will be because they did not feel a part of the process of 
development or were not consulted. The perception of "ownership" is powerful (Thurskya and 
Mahemoffb, 2007). Users may also try the tool only once, find that they don't understand it (or 
it doesn't fit the need, or it returns an obvious response) and stop its use, or just never try it at 
all. Therefore, within this Objective we must address a) actual use of the tool by the target 
group, and b) determining the optimal expected response after delivery of the solution to the 
recipient. Just to clarify, this objective is not intended to replace the participatory design/user 
interface development step which would naturally be completed by the solution developer. 
However, it does bring together and integrates the voice of the consumer into this critical stage 
of the process and would be a strong and important partner in this process. 

Actual Use (Uptake) of Tool & Post-Use Expectations (a & b above): As Moxey et al. (2010) 
describe, simply providing a tool, no matter how useful, does not result in uptake by the 
intended users. Workers can resent being directed to do certain tasks around something that is 
new without having input. Therefore, our approach to addressing both of these issues will be 
engaging the right staff at the right level. This fits directly with how Allegheny County 
traditionally elicits feedback from its stakeholders (e.g., through year-round input and participation in 
committees, work groups, panels, public meetings, surveys, data collection activities advocacy groups, 
stakeholder-specific groups, and other ways of communicating ongoing feedback to the County, such as the 
Children's Cabinet [Allegheny County Mental Health Plan, 2012: p. 4; Children's Integrated Service Plan, p. 2]). 
This process is much the same as described in Objective 1 and the process at the end of 
Objective 3. However, by this point in the process we will become much more familiar with the 
AC DHS data, players, and the playing field. Because of this, we may or may not necessarily 
need focus groups for discussions in all circumstances but could solicit feedback through 
electronic surveys. This format could occur one time, but would actually be the start of a 
process. After receiving feedback from the first survey, we would build out a potential 
prototype mock-up using graphics or business intelligence software (e.g.. Tableau) and put out 
results for votes and feedback. Depending on the environment, we may suggest something to 
engage staff such as running a contest for best design suggestion. This would bind people to 
the project on multiple levels while preparing/priming them for the real build which will follow. 
Part of the intent will be to build excitement and anticipation around the tool to enhance the 
uptake potential. 
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Post-use expectations (of AC DHS) for users of the decision support tool(s) will be tied in 
to what the users (themselves) are expecting. Users will want value. The organization will want 
results. While these goals are ultimately compatible, they must achieve sufficient concordance 
to achieve the outcomes desired by all. We will pull together a design (not focus) group of 
frontline workers and administrators who will develop a set of common expectations of how 
the tool should be used and how users should react to the decision support presented. We will 
use these suggestions to work with the software/tool developers to include in the design as, at 
least, a starting point. 

Use Scenarios: We intend to focus on developing responses to the following 2 questions: A) 
"How can we assist workers to use the data collected on client (family) risks, strengths, and 
needs to target interventions (including specific evidence-based interventions)?", and B) 

"Which clients receiving behavioral services may not be receiving services at the appropriate 

level of care?" The figure 
at left demonstrates the 
discovery process 
necessary to examine the 
nuances of this question. 
Starting with Objectives 1 & 
2, the immediate issue is to 
determine the target 
condition which will be the 
object of the target 
intervention. There are 
then the questions of the 
use condition(s) (could be 
one, some, or all) and the 
types of intervention styles 
(discussion of delivery 
methods would occur later 
via Objective 4). Data 

inputs would be considered next and would involve the Risks, Strengths, and Needs noted in 
the original question but also a host of other mediating factors that would determine: the 
"dose" of the intervention, whether to factor in the influence or contribution of other agencies, 
whether the client is receiving additional services now which would continue along with the 
intervention, and more. Outcomes would be defined at the outset of this process but revisited 
at the end so that the intent and focus remain consistent. This entire process just described 
would occur via focus group discussion and then distillation and analyses of qualitative data. 

The results of these analyses would prompt the start of the work of NYU to begin the Expert 
Review and Input phase. Once complete, we would determine which method of predictive 
modeling would be most appropriate for these data (Objective 3) and this situation and then 
conduct those analyses. We would then return the results and interpretation of the analyses to 
the focus group for review and validation as well as determining the types of output 
recommended and the action expected from accessing the data (Objective 4). We would use 


Question A; How can we assist workers to use the data collected on client (family) risks, strengths, 
and needs to target interventions (including specific evidence-based interventions)? 


OBJECTIVES 1 & 2 


Determine Target Condition 
Requiring Intervention 


AC DHS Staff/Frontline 
Expert/Consumer Focus Groups 


Determine Use Conditions : 

In Field? 

In Office? 

Group Intervention? 
Individual Intervention? 
Requires Materials? 


Determine Use Type 
Self-Help? 

Quick intervention? 
Worker-administered? 
Other intermediary 
(police officer, etc.)? 


Determine Inputs: 

Risks, Strengths, Needs 
ALSO CONSIDER: 

Resources 

Current interventions 
Characteristics 
-Demographics 
•Geography 

-Medical, mental health condition 
Intervention intended for: 

-Client alone? 

-Client * Family? 

-Family alone 


Other agencies Involved? 
Other agencies needed? 


Determine Target Conditions 
Outcomes 


Expert Review and Input fNYUl : 
-Additional inputs, independent 
variables 


OBJECTIVE 4 


Create Model(s) 
Determine 
appropriate 
predictive model 
analysis method(s) 
to use 

Perform Analyses 


Post-use 

expectations and 
planning. 

Advise software 
builders around 
these design 
elements. 


Process follows description In Project Diagram and Timeline. 


CCSI-NYU: Decision Support Tools and Predictive Analytics in Human Services. 23 




this information to work with the software developers to shape their product build, in 
particular the delivery venue and considerations around output. 

The second question would follow an identical process, but the factors within Objectives 
1 & 2 would be different. As shown in the figure for Question B below, while the process is 

similar to that outlined for 
Question A, the work 
would be very different. 
The question posed here 
requires a definition for 
every key word. Those 
definitions would be based 
on the intent of the 
purpose, such as incidence 
vs. prevalence (e.g., is this 
for someone who has been 
involved in the system and 
is showing an increase in 
symptoms? Is this from a 
static review of warehouse 
data to flag individuals for 
follow-up? etc.). The focus 
group may find that the 
true intent actually requires a different question which could then change all of these inputs. 
Following finalization of these processes, the process flow would return to that outlined for 
Qbjectives 3 and 4. 

3. Describe how the program will be integrated with other information technology and tools: 

It will be imperative that all participants regardless of role are familiar with the AC DHS 
Data Warehouse. The Data Warehouse is expansive, containing 650 million records on nearly 1 
million individuals. It holds data from 29 internal programs within the large program categories 
and is also actively involved in a number of collaborations with external sources, such as city 
schools and the US Department of Census. AC DHS also has several targeted processes of 
collecting data through software applications. All of these sources can contribute elements to 
predictive modeling and will be integrated into the discovery process. When building an 
application such as one intended to support decision-making, the most basic element is to 
ensure that the targeted data are available to use. The data in AC DHS reside in the data 
warehouse, which generally operates as the name implies: it warehouses data as its inventory. 
The "inventory" can be accessed by knowing where the specific required data are in the 
warehouse and how to reach them. The AC DHS data warehouse is built on a structured query 
language (SQL), Qracle platform. Accessing the data through the simplest means might be to 
construct SQL queries. Qne or many data subset(s) for use in predictive modeling or, later, for 
decision support, can be extracted using SQL queries. These can be automated to run at a 
certain time of day at a certain frequency making the data available for the tool without a drag 
on the system operation, generally taking place on a dedicated server. This type of approach 


Question B: Which clients receiving behavioral services may not be receiving services at the 
appropriate level of care? 


OBJECTIVES 1 & 2 


Data cut: individuals receiving 
behavioral health services ONLY 


AC DHSStaff/Frontllne 
Expert/Consumer Focus Groups 


Determine Use Conditions: 
Age group? 

Diagnosis (or Dx Category)? 

Define/Oelineate "Services" 
Medications? Inpatient? 

Preventive? Crisis? 

Rehabilitation? Clinic? 

Stabilization? CBT? 

Support? Multiple? 


Define/Oelineate "ADDroDriate": 

Billable? 

Evidence-Based Practice? 

Expected vs. Target vs. Baseline? 

Other conditions: Physical issues impacting mental health? 
Other factors: Is current level actually appropriate but needs 
adjustment rather than increase (e.g., meds stopped working 
and need a different type). 


Deflne/Dellneaf "Level of Care** 
Inpatient vs. Outpatient? 
Residential vs. SNF? 

Other considerations? 


Determine Target Conditions 
Outcomes 


Expert Review and Input tNYLil : 
-Additional inputs, independent 
variables 


OBJECTIVE 4 


Create Model(s) 
Determine 
appropriate 
predictive model 
analysis method(s) 
to use 

Perform Analyses 


Post-use 

expectations and 
planning. 

Advise software 
builders around 
these design 
elements. 


Process follows description in Project Diagram and Timeline. 
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would be compatible with the AC DHS warehouse. Of course this simple way is only one of 
many that could be used to perform similar data retrieval. 

A software build would access the data, but would have a series of rules and reporting 
layers depending on the mechanism. One such use scenario is presented in the last section of 
Appendix A for OBP. There we used Presentation, Persistence, Data Injection, Business Logic, 
and Build layers. The Presentation layer used JSF to display the data via a web browser. The 
Persistence layer used Hibernate configuration files to map Java objects. We used backing 
beans for data calls and added our own business rules constructed from the literature and 
expert input and used Maven for the build. Here we assume that the tool would operate 
outside of the current system but could be called in at certain points automatically or offered as 
an option via a prompt. Regardless of the scenario, it would be important for the software 
builder to work with our team to do a thorough audit of the AC DHS system capabilities and 
understand the intended receiver of the decision support tool. One such (obvious) example 
would be whether the output would need to be seen in just a mobile format or must be 
scalable to a number of different output venues. 


4) Provide a timeline for design and development: 


PROJECT DESIGN 
AND TIMELINE 


Building Model Components 


Objective 1 


Objective 2 

Define Outcomes 


Define Predictor 



Variables 



QUESTION A : 

Month 1 
QUESTION B : 


Purpose: Succinctly define 
outcome variables via expert 
input. 

Purpose: Determine all practice- 
related predictor factors/ 
variables: expert input. 



Process Method: FOCUS GROUPS 

Participants: AC DHS Staff/Experts 
Conveners: CCSI (NYU observe) 

ALSO: Ensure/Confirm sufficient quality of intervention. 



TIMELINE 

Month 2 
Month 3 


Building Model Analyses 


objective 3 

1. Confirm/validate 
all model 
components (Lit. 
Review, experts) 
Conveners: NYU, CCSI 


2. Build and analyze 
predictive analytic 
model. 

Data Analytics: CCSI, 
NYU review/advise 


3. Produce output 
report and roadmap/ 
blueprint for Decision 
Support Pathway. !. 
Data Analytics: CCSI, 
NYU review/advise 


Month 3 
Month 4 


Implementation 


objective 4: 

Purpose: Solicit and integrate 
review and alterations of model. 


Process Method; FOCUS GROUPS 

Participants: AC DHS Staff/Experts 
Conveners: CCSI {NYU observe) 


1. Confirm model 
and results 


2. Actual Use 
Projections: Uptake 
issues/solutions 


3. Post-use 
expectations planning 


^ 

Advise software 
builders around these 
design elements. 


Months 4-12 
Months 5-12 


We present a diagram of the proposed project roadmap and timeline. CCSI plans to begin with 
the kick-off meeting to start the project at the beginning of Month 1. CCSI will then spend that 
month formulating our planning and working with our AC DHS contact staff to arrange the 
focus groups necessary to address Objective 1 for Question A. CCSI will spend Month 2 running 
the focus groups and preparing and analyzing the qualitative data for use by NYU which will 
start its refinement process immediately. During Month 3, NYU will finish its work and both 
CCSI and NYU will finalize the predictive model structure, determine the appropriate analyses 
methods, conduct the analyses, and interpret the results to bring back to the focus group for 
confirmation. The work of Objective 4 work will begin immediately as we engage with the 
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software design/build group(s), which will continue for the remainder of the year. We will 
stagger the approach to Question B by one month in order to allow time for the work on 
Question A to progress. We acknowledge that there is substantial activity in the beginning 
months. The reason we are pushing hard early is to allow enough time for us to get this 
important design information to the software build group(s) so that they may have a completed 
project by year-end, as required in the RFP. 

Sustainability : 

The actual process of determining potential areas for decision support and narrowing 
down to clearly defined elements has all of the features of a CQI process, to which ideally this 
project will contribute for AC DHS. The quality of the features recommended through the focus 
group process should add sufficient value to the final software product that it will enhance its 
marketability and applicability to other counties and large provider organizations. 

5) Provide examples of tools/systems already developed: 

We present examples of three systems already described in above sections: a) Performance 
and Contract Management System (PCMS), b) Addiction Recovery Employability System (ARES), 
and c) Qutcomes-Based Prescribing (QBP). Detail is provided in Appendix A. 

6) Describe the staffing plan. (Aligns with project plan description above and Budget Narrative) 
For CCSI: John F. Crilly, PhD, MPH, MSW: Principal Investigator, will oversee all aspects of the 
project and will design the predictive models, analyses, and interpretation. 

TBD: Project Coordinator: Will work directly with AC-DHS staff to construct and conduct all 
focus groups, will assist with analysis of qualitative data, and will be responsible for timeline. 
TBD: Data Analyst: Will run all analyses in the designated software packages. 

For NYU: Andrew F. Cleek, PsyD: Co-investigator, will oversee all aspects of the project at NYU 
and work in close collaboration with Dr. Crilly around predictive model design and analyses. He 
will be aided by Micaela Mercado, PhD who will lead the Expert Input process. The NYU project 
will be coordinated by Ashley Fuss, MSW who will interact closely with CCSI and AC DHS, 
including involvement in AC DHS focus groups. 


REFERENCES 


CCSI 

David Putney, MA, CASCAC, Director Monroe 

Co. Qffice of Mental Health 

(585) 753-6047, doutnevPmonreocountv.gov 

NYU-McSilver 

Elisa Chow, PhD, LMSW 

Independent Community Living, Director 
212.385.3030 ext 16005, echow(®iclinc.org 

Ellery Reaves, MPA, Commissioner 

Erie and Genesee Co. Dept. Mental Health 
(716) 858-8531. Ellerv.Reaves(®erie.gov 

Wendy Brennan 

Executive Director, NAMI-NYC Metro 
212.684.3365, wbrennan(®naminv.org 

Arthur R. Johnson, LCSW, Commissioner 

Broome Co. Social Services and Mental Health 
(607)778-2600, aiohnson(®co.broome.nv.us 

Marty Forth, LMSW 

Senior Executive, YMCA of Greater New York 
212-630-9647, mforth(5)vmcanvc.org 
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3. BUDGET AND BUDGET DESCRIPTION 

Budget 


County of Allegheny, Pennsylvania 

Department of Human Services 

Decision Support Tools and Predictive Analayses 

3. Budget (excel) and Budget Description (word) 


Line 

Item Budget Revenue 

1 Anticipated revenues including any matching funds 

Total Revenue 


Extended Price 


Line 

Item Budget Expense 

2 Personal services and associated agency costs 


Agency 

CCSI 

NYU 

NYU 

CCSI 

NYU 

CCSI 


Position Title 


Employee Name 


Extended Price 


Principle Investigator John Crilly, PhD, MPH, MSW 
Co - Investigator Andrew F. Cleek, Psy.D. 

Ashley Fuss, MSW 
To Be Named 
Micaela Mercado, Ph.D. 

To Be Named 


Project Coordinator 
Project Coordinator 
Evaluation Staff 
Data Analyst 


Subtotal Personal services and associated agency costs 

3 Travel and associated costs 

Mileage and related ground transportation expenses 
Per Diem Hotel and Meal related expenses 

Subtotal Travel and associated transportation costs 

Total Expense 




Total Funding Requested 


Budget Narrative/Description 

1. There are no anticipated revenues as shown in Line Item 1 - Budget Revenue. 

2. Line Item 2 outlines budget expenses including personal services and associated agency 
costs. There will be three staff from Coordinated Care Services, Inc. (CCSI) included on the 
project and three from NYU-McSilver, described as follows: 

• John Crilly, PhD, MPH, MSW : Principle Investigator. Dr. Crilly's role will be to oversee the 
entire project. Although he will work closely with Dr. Cleek in these areas, he will be 
responsible for overseeing the analysis of qualitative data, development and analyses of the 
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predictive model, and all interpretations resulting therefrom. He will be on site to 
participate in the focus and design groups and assist in the cross-walk collaborations both 
with NYU and the software developers. 

TBD : Project Coordinator. The CCSI Project Coordinator will be a Masters-trained, senior- 
level CCSI employee who has experience in the arenas of direct care, program evaluation, 
and research with a strong understanding of data and technology. This PC will be 
responsible for interactions with AC DHS staff to plan logistics in the AC DHS environment 
and develop and implement focus groups. This PC will run all focus groups and will be 
responsible for the recording and analysis of the qualitative data in conjunction with Dr. 
Crilly. This PC will be responsible for maintaining project adherence to the timeline. 

TBD : Data Analyst. The data analyst will be a Masters-trained Sr. Programmer/Analyst who 
will be proficient in the statistical software and analyses methods necessary to conduct the 
work in this proposal. This will include SPSS, SAS, Stata, R, and Atlas (qualitative data 
analysis software). 

NYU-McSilver staff will include: 

• Andrew Cleek, PsyD : Co-Investigator. Dr. Cleek's role will be to oversee the Project as it 
occurs at the NYU-McSilver site. Dr. Cleek will assist in gathering experts for input into the 
predictor and outcome variable designations and expediting the process. He will 
collaborate with Dr. Crilly in the development, analysis, and interpretation of the predictive 
models. He will also assure smooth transitions of processes requiring crosswalks between 
NYU-McSilver and CCSI, AC DHS, and software developers with Dr. Crilly. 

• Micaela Mercado, PhD, LMSW : Evaluation Staff. Dr. Mercado will facilitate and oversee the 
operation of the Predictor/Outcome variable determination at NYU. Working with Dr. 

Cleek, she will design the appropriate group formats and recruit panels of experts to 
provide input into the features of Questions A and B in this proposal. 

• Ashley Fuss, MSW : Project Coordinator. Ms. Fuss will be responsible for the operation of 
the project for NYU-McSilver. She will work closely with CCSI and AC DHS and will 
coordinate all aspects of the project at NYU-McSilver. She will be responsible for ensuring 
adherence to the timeline. She will also be involved as a note-taking participant in the AC 
DHS focus groups to ensure an adequate link between the work conducted by Dr. Mercado 
and the input provided by the focus group members. 

3. Each staff member is individually listed with their fully loaded compensation rate which 
includes costs associated with project time, agency administrative time, vacation and holiday 
time including mandated and non-mandated benefits as well as any associated office supplies, 
office equipment, infrastructure and agency administration and overhead. 

4. An estimation of their time required for this project is listed under the FTE column, which is 
the Full Time Equivalent based on a 40 hour work week. Work tasks included in this estimation 
are outlined above. 

5. The Extended Price column is calculated by multiplying the fully loaded compensation rate 
by the FTE. The Line Item 2 Subtotal is 


CCSI-NYU: Decision Support Tools and Predictive Analytics in Human Services. 28 




6. Line Item 3 summarizes travel and associated costs. Mileage and related ground 
transportation expenses are calculated along with per diem hotel and meal related expenses. It 
is anticipated that the Principal and Co-Investigator will travel to Allegheny County on a bi¬ 
monthly basis, 2.5 days per trip. Trip duties will include attending focus group meetings, 
meetings and consultation with AC DHS staff, ensuring overall operation of the project and, 
later, meeting with software developers. The CCSI and NYU Coordinators will have a total of 14 
trips, 2.5 days per trip, to be distributed throughout the year as needed for conducting focus 
groups and attending meetings with AC DHS staff at all levels. Mileage was calculated using 
round trip miles and $0.56/per mile mileage reimbursement. The US General Services 
Administration figures were used to calculate per diem travel costs. Greater detail for each 
staff member is listed below. The Line Item 3 Subtotal is 


Mileage and other travel 

Total 

1 staff NYU: 14 trips for 2.5 days each 


Co-lnv NYU: 1 time bi-monthly for 2.5 days 


1 staff CCSI: 14 trips for 2.5 days each 


Princ Inv CCSI: 1 time bi-monthly for 2.5 days 


Total Mileage and other travel 

$ 


Hotel, meals, etc. 

Total 

1 staff NYU: 14 trips for 2.5 days, 2.5 nights 


Co-lnv NYU: 1 time bi-monthly for 2.5 days 


1 staff CCSI : 14 trips for 2.5 days, 2.5 nights 


Princ Inv CCSI: 1 time bi-monthly for 2.5 days 


Total Hotel, meals, etc. 

$ 
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APPENDIX A 

A. Performance and Contract Management System (PCMS) 



Contract Management System 


A Web-based tool for oversight organizations 
to manage service contracts and estabiish 
performance accountability and evaiuation 


Feature 

Benefit 

Web-based application 

• Minimize hardware for easy access 

• Eliminate need for cumbersome local updates with 
upgrades on a central server 

Provider portal ~ Home page summary view 

• See status of contracts at a glance 

• View a quick list of delinquent events and deliverables 

• Manage to-do lists 

• Link to standard reports 

• Link to data collection forms 

• Collaborate on documents and spreadsheets with 
shared folders 

Contract Tree summary view 

• See all program- and service-level events and 
deliverables in one view 

• Drill down to the details of each linked item with one 
click 

• See contract information related to critical dates and 
funding 

Shared contract view for oversight agency and providers 

• Create efficiencies by eliminating questions about 
contract status and details 

User-friendly functions to create multiple contract 
structures and requirement sets 

• Set up contracts, deliverables, and events efficiently 
and logically 

• Create customized contract templates 

• Create and define unlimited data collection fields 

• Renew contracts easily 

Workflow management (partially developed) 

• Track internal processes pertaining to contract life cycle 

Automatic audit trail 

• Automatically track changes for internal and external 
accountability 

Automatic e-mail reminders (in development) 

• Take the burden out of remembering to take action on 
events and deliverables 

• Allow customization of e-mail text as well as definition 
of conditions and timing for distribution 

Note fields 

• Allow “in-system” communication between users 

regarding specific deliverables or contract components 

Performance data capture 

• Provide an easily accessible way for any provider to 
submit data 

• Collect data in a more usable format than paper 
reporting 

• Provide the opportunity to standardize performance 
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measures among users 

• Serve as a data store, allowing providers access to 
their data 

Data warehouse - ad hoc reporting capabilities 

• Allow unlimited analyses with Web-based query and 
reporting tools 

• Isolate data for analysis of a specific element across 
services 

• Isolate multiple data elements for analyses from a 
single provider 

• Create and save common formats for easy periodic 
reporting 

• Extract data to spreadsheet for additional analysis, 
benchmarking, graphical display, etc. 


Screen shots of PCMS are as follows: 


Sifin-ln Screen 


MANAGEMENT SYSTEM 




Ptflonnwcf Rvpomng 


UAWMCNU 


fteports 


Ooctinwiift 


Work Screens (followiriR paRe): 
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B. Addiction Recovery Employability System (ARES) 


ARES 

ADDICTION RECOVERY EMPLOYABILITY SYSTEM 


What is ARES? 

ARES - (Alcohol Recovery Employability System) is an Internet-based computer application that is 
designed to link the Monroe County Department of Health and Human Services and Community 
Substance Abuse Providers for the purpose of monitoring the treatment progress of individuals receiving 
Social Services benefits. It is accessed via an internet connection over a secure Virtual Private Network. 
ARES enables direct, secure internal Email, instant evaluation scheduling, smart tasks to be sent to users 
at key times, and is administered by each user organization independently. It makes reporting of 
monthly attendance information and 60 Day Progress Reports fast and efficient for Providers and sends 
the information directly to the desktop of the DHHS Examiner who is dealing with the case. It receives 
daily updates from the Welfare Management System and provides current information to those who 
need it without accessing two systems. 

Who Participates? 

• The Department of Human and Health Services, Division of Social Services - 

o Temporary Assistance, Medicaid and Employment Services 

• Fourteen (14) Community Substance Abuse Providers 

• A contracted Behavioral Health Care Professional 

• Coordinated Care Services, Inc.(CCSI) as the System Manager 

• Innovative Solutions - System Designers 

How was ARES developed? 

Discussions were held to survey the needs of Health and Human Services staff, the Provider 
Community, CCSI, and Innovative Solutions to determine the elements of a cost effective, real-time 
alternative to the previous manual process. The focus was on ease of use, flexibility, and utility for all 
the participants. Operational Specifications were determined and the design specifications of the 
application were developed and refined. A test system was developed and the full system was 
completed and tested. Training was provided to the User community. Data from the previous Chemical 
Dependency Tracking System was migrated to the new application and the system was implemented. 

How does ARES work? 

Public Assistance and Medicaid applicants and recipients are screened at DSS to determine if 
they might need help in addressing a substance abuse problem. If the screening indicates a problem, 
the DSS Examiner enters the client demographic information into ARES and makes an on-line evaluation 
appointment with a participating community provider. The client receives a printed confirmation of the 
appointment noting directions to the provider and cancellation policies. Clients are expected to return 
to their previous Provider if they received treatment in the past. 

Once the provider conducts their Chemical Dependency evaluation, they note the results in the 
ARES system and if the client needs to attend treatment they request approval for the level of care that 
they deem appropriate from the Contracted Behavioral Health Professional. On behalf of the County, 
the professional reviews all level of care plans and notifies the CD Provider of the DHHS position to 
support or not to support the treatment recommendation. Requests for Inpatient treatment require a 
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detailed clinical justification for that level of care since it is very costly. Treatment Providers must also 
request approval for a transfer of Providers. Once Clients have their treatment approved and begin 
attending a treatment program, the Community Provider enters their attendance in ARES each month 
and completes an on-line progress report every 60 days. CD Providers note the client's employability 
status on their progress reports. ARES participants are referred to the DSS employment unit once they 
are determined to be able to seek employment. 

What are some of the useful features in ARES? 

• ARES is a "smart" system and has been programmed to recognize key points in case processing, 
identifying who needs to know this information and then sends the information automatically. 
For example, if a client is discharged from treatment, a notification is system generated to the 
appropriate Examiner alerting them so that the case can be reviewed and either closed or 
motivated back to treatment. Notifications are generated to DSS and Provider staff as well as 
the Behavioral Health Professional. ARES has on-line evaluation scheduling at any participating 
Provider and the system reviews the client record to help determine the most appropriate 
Provider based on case history. 

• ARES has secure, internal e-mail which connects all users in the system 

• ARES connects with the Welfare Management System daily and when a case closes at DSS and 
the effective date of closing is reached, it closes the ARES tracking episode and notifies all 
parties dealing with the case. 

• ARES provides Users templates for the reporting of evaluation and treatment recommendations, 
attendance information, progress reports and discharge information. 

• ARES supplies a full range of reports to Users. Reports have been designed to help keep track of 
daily operational processing, provider information on user performance and to gather 
information for additional analysis. 

What does ARES look like? 

ARES is designed as a typical windows based, user-friendly, graphically based interface. The 
screens are arranged so that the user is intuitively led from one to another and has proven to be 
relatively easy to learn. Below are examples of some typical screens in ARES: NOTE-AIIpatient 
information is dummy data. 

Evaluation Scheduling Calendar 


3i Schedule Client Evaluation - Microsoft Internet Explorer provided by Compaq 




Edit yiew Favorites Tools Help 

Welcome Nick Dss, 
Active Client: Maidi 



! 1 unread message(s) and 5 task(s). 


Schedule C/ier7t Evaluation ^ 
Choose Provider: | xxprovider 



ll!OOAMfSl 


Appointment: (Note: if 
1 12/18/2002 n :OOAM~ 

Notes: (AddJ^atesJ 


nuaiiy - ex: 01/01/2002 05:OOAM) 
ish Speaking 


I Schedule Appointment ] 



ices. 34 







































































Client Main Page 


3 Client Episode Information • Microsoft Internet Expbrer provided by Compaq 




File Wit View Favorites Tools Help 

ARES 



Client 


Home Client DSS Messages 


Welcome Nick Dss, you have 1 unread message(s) and 5 task(s). 
Active Client: Wilson, Brian 


EDisode-10/17/2002 


Client Episode Informetion ** 


I episode found 


View Episode 

A 

Episode Start Date 

T 

A 

Episode End Date 

T 

A 

Episode Status 

▼ 

Iview 

10/17/2002 1 |Open 


SSN 

300000002 
Cin # 

DH22222Q 
Actual Start Date 
X2/01/2002 
Responsible Agency 
xxprovider 
Attendance Report 
Dec 2002 

Episode Close Date 
Notes (Add Notes) 


DOB 

02/02/1982 

DSS Unit 
24 

Projected End Date 

01/31/2003 

Responsible Counseler 
Current Program 
Episode Close Reason 


DSS Case # 

BA2222 

DSS 

Nick Dss 

Actual End Date 

Progress Report Date 

12/09/2002 

Evaluation Appt Date 
12/03/2002 


12/2/2002 12:04:22 PH From: Honitor, Annette 
Expecting Brian to need Tx 



Attendance Summary 

^ ARES - Client _ ^ 


Home Client DSS Messages 


Vi^elcome Nick Dss, you have 1 unread message(s) and 5 task(s). 
Active Client: Wilson, Brian 



Client Attendance W 
1 record found 


▲ 

Date 

1 ▼ 

A 

Provider 

▼ 

A A 

Program Level of Care 

▼ ▼ 

A 

Scheduled 

▼ 

A [ * 

Attended Excused 

▼ 1 ▼ 

|Dec-2002 

xxprovider 

Outpatient 

fl2 

fio [2 



Notes: (Add Notes) 
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What Types of Information is Available on ARES? 

ARES holds a complete treatment history (currently back two years) of each client, records of 
attendance at various programs and copies of a Progress Reports completed by any Provider they have 
received services from. It holds records of the clinical justification and background information for each 
Inpatient referral made. ARES has a daily direct link to the New York State Welfare Management System 
(WMS) which is the computer system that is used to identify eligible individuals and to authorize Public 
Assistance, Medicaid and Food Stamps benefits. Specific eligibility elements such as including address, 
case and individual eligibility status, the client's primary language and their veteran's status, are 
imported and ARES is updated with new information. Providers can verify their client's DSS status 
without consulting EMVES (electronic Medicaid Verification Eligibility System). 

Additionally, the client's DSS employment activities are listed on the ARES system. This 
information is also imported from the WMS Employment Subsystem. The CD treatment schedule is 
entered by the Provider and efforts are made to reduce interference with treatment when employment 
activities and DSS appointments are scheduled. Information about Other Programs that the client is 
involved with can also be noted in ARES. These would include Methadone Maintenance, Residential 
Living, Case Management Activity, Criminal Justice involvement, and Mental Health Treatment. If the 
client's case is closed at DSS that information is available through ARES with an explanation of the cause 
of the closing and sanction if applicable. 

What about Confidentiality? 

Innovative Solutions, which designed the ARES program, was very careful to ensure that all 
information in the system would conform to HIPAA (Health Information Portability and Accountability 
Act) Standards and designed in compliance with OASAS confidentiality requirements. In general the 
system only allows access to an individual client's information by their specific treatment provider, by 
Health and Human Services Examiners and by the Contract Service Agency. Viewing or changing a 
client's information is restricted by the ARES security system of organization registration and by use of 
individual roles, user sign-on, and passwords. 

What Outcomes will be measured from ARES? 

The Monroe County Department of Health and Human Services and the Community Providers 
have all signed a Memorandum of Understanding (MOU) that outlines the expectations for the ARES 
project. The MOU is updated annually. Some of the expectations that are measured include: 

• Guaranteed Five (5) day evaluation appointment availability 

• Complete Evaluation within two (2) days of the appointment 

and include an opinion as to the potential employability of the client 

• Start the client in the approved treatment program within 10 days of the 
evaluation scheduling date 

• Enter Attendance for all participants by the 10*^ of the month 

• Complete Progress Reports every 60 days 

A Monthly Process Report is produced to summarize Provider compliance with these 
expectations. DSS has agreed to provide referrals to the participating providers and to guarantee 
payment for any DSS arranged evaluation. ARES provides a full range of operational reports to assist in 
daily activities and in the analysis of data to determine effectiveness and timeliness of treatment and 
reporting. Cost containment is also a valuable product of ARES. Clients not participating in mandated 
treatment are quickly identified so appropriate action can be taken. Review of proposed treatment 
plans by the Behavioral Health Professional allows some control over Inpatient or Out-of-County 
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treatment. Information about client's who have multiple treatment histories and make little progress 
can be identified and referred for case management interventions. 

What else should we know about ARES and Outcomes? 

The ARES Partners - DSS, Community Providers, Service Agencies, and CCSI meet monthly to 
discuss the program and to discuss improvements in the system along with other treatment or process 
issues related to the effective community approach to Chemical Dependency Treatment. This 
community approach to the CD treatment process has been very helpful to Providers and Health and 
Human Services in providing a real time, technology based solution to assist in the management of 
chemical dependency services and its volatile population. 

ARES has been shown to be an effective tool in monitoring client compliance with prescribed 
treatment. Clients enter treatment sooner and the no show rate for evaluations has been improved 
making management of resources by Providers easier. Ease of reporting and accountability has 
improved Provider compliance with reporting of attendance and Progress in treatment. Providers 
report that clients who do engage in treatment stay in treatment longer improving their chances for a 
productive and stable recovery. Inpatient costs have been significantly reduced and although 
Outpatient costs have increased minimally, the net savings is significant. The partnership established 
with local providers has minimized the County dollars leaving the community for treatment providers 
elsewhere. Communications have been facilitated between the Human and Health Services staff making 
for more coordinated efforts resulting in better, and more timely treatments for chemically dependent 
individuals and most importantly improving the chances for a stable recovery, return to the workforce 
and self-sufficiency. 


CCSI-NYU: Decision Support Tools and Predictive Analytics in Human Services. 37 




C. Outcomes-Based Prescribing (OBP) 

The following is a draft of this paper, currently in the publication process. Please do not 
distribute beyond the use of this RFP review or use any part at this time beyond the current 
scope. 

Outcomes-Based Prescribing: A Novel Use of Electronic Health Record Data 
To Inform Patient and Clinician Treatment Choices 

John Crilly, PhD, MPH, MSW 
Lan Chi T. Luu, PhD 

VISN 16 MIRECC 

Southeast Louisiana Veterans Healthcare System 
Coordinated Care Services, Inc. (CCSI) 

The authors would like to extend their great appreciation to others who played supportive roles in this 
venture: Diane Niemann, VISN 2 CIO, James McCain VA Central Office - Innovations Program, W. Ashley 
Byrd of ICF International which built the software, Joseph Linza of CWI Contractors, Jeff Wolfganger of 
Canandaigua VA, Jane Wood and Krista Stephenson of the VISN Center of Excellence for Suicide 
Prevention. Journal: British Journal of Medical Informatics 

Abstract 

General information about how medications work and their side effects can be readily accessed by 
clinicians and patients through abundant online resources. While access to these resources is important 
for patients, the applicability of general medical information to the individual's own experience remains 
limited. This gap impedes the process of better engaging patients in decisions for their own healthcare. 
Personal healthcare support and tools exist and there is no shortage of individual healthcare data. 
However, tools are lacking to bring personalized healthcare to life. Using a concept of Individualized 
Comparative Effectiveness Research, we developed a working prototype of a practical tool which draws 
upon enterprise electronic health record data to help patients and clinicians better predict individual 
outcomes when making decisions for new or additional treatments. 

Introduction 

Treatment methods for physical and mental illnesses vary widely across the US (Wennberg, 

2004; Baiker et al., 2004; McGlynn et al., 2003). Patient outcomes also vary and are dependent not only 
on area treatment and practice methods but other important factors such as geography (e.g., climate, 
region, culture) and demography (age, race, ethnicity, gender, language, culture). In general, we know 
that treatments and medications currently on the market work to some extent based on the results of 
clinical trials and population-based public health studies. One of the recognized weaknesses of these 
studies is that while findings are applicable to the general population, they cannot necessarily be applied 
to any individual case. Typically, individuals receiving new medications are put through their own 
rounds of medication trials until one or a mix is found to have sufficient clinical impact (Alemi et al., 
2009). At such a point, subsequent trials cease until a new medication is needed, and the cycle is 
repeated. This trial-and-error process is both inefficient and expensive for the healthcare system and 
the patient. Within the past decade patients and healthcare systems have begun to develop ways for 
individuals to share their own personal experiences with treatments and medications. Efforts include 
self-report tools (e.g., the Medicare Health Outcomes Survey) and Internet-based self-report forums 
(e.g., www.PatientsLikeMe.com ) as well as personal health records (PHRs) offered by healthcare 
systems and online public access (e.g., Microsoft Healthvault). 
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The game-changer for this problem is increasingly robust electronic health record (EHR) systems 
and EHR data. EHRs are slowly being adopted by private physician groups and practices (Hsaio et al., 
2009; Jha et al., 2009; Blumenthal and Tavenner, 2010) and their aggregate data can begin to serve 
individual patients in addition to the overall corporate structure. Innovative tools can boost the 
movement toward the development of "true" personalized medicine. This paper describes a tool 
prototype built using model data from the VA's VistA EHR system. This tool can provide the ability for 
individual patients to seek clinical outcomes information from treatment/medication experience of 
other patients with similar characteristics at the local level. 

Background 

The concept of "personalized medicine" is central to healthcare reform policies. Briefly, the 
mission of personalized medicine is to deliver the right care for that individual, to that individual, a 
matter of more precisely tailoring treatments at the individual level based on data. One increasingly 
ubiquitous platform through which this may be accomplished is health information technologies. 
Technologies to deliver health information are abundant: conventional popular media (radio, 
newspaper, television), academic literature, online informational websites, FDA websites, manufacturer 
recommendations, internal health system prescribing guidelines, insurance company regulations, and 
summary knowledge provided by EHR systems regarding medication safety and efficacy are some of the 
more popular avenues. The Internet and its components are the most important platforms for 
promoting greater involvement of patients in maintaining their own health. For example, patients can 
now share feedback and commentary from their own personal illness and treatment experiences with 
other patients through web sites such as www.PatientsLikeMe.com,www.AskaPatient.com, and 
www.crazvmed.us . Other sites such as www.webmd.com and www.revolutionhealth.com provide 
more formal, science-based information about illness and treatments. 

The content in both patient commentary and formalized informational websites can help 
patients make choices about their treatment and better anticipate treatment outcomes (Hughes and 
Cohen, 2011), but only to a point. HIPAA laws allow only certain de-identified fields, generally age, 
gender, diagnosis, treatment, and symptoms, for use to "drill down" into the data. For example, a 
patient in rural Louisiana cannot know that the feedback she is receiving from diabetes medication 
reviews is from individuals in her own part of the country, and from patients living in rural areas as she 
does. Patients and providers cannot drill down deep enough into the data to determine how patients 
fare within or between geographic areas (like Des Moines, Iowa vs. New Orleans, Louisiana), or even 
within the same healthcare system. This limits more robust uses of these kinds of data. 

Patient personal health records (PHR) have received increasing attention as useful 
communication and health maintenance platforms which can be shaped to help these problems. 

Beyond being simply portals, PHRs allow patients to interact with parts of their own healthcare record 
and their healthcare provider (Page, 2010). They can also host information-based healthcare and 
wellness tools. While intrinsically useful, PHRs have had a slow uptake among patients and providers 
(Rhodes, 2007; Markle Foundation, 2011). Adding useful features and tools may not only help bring 
more patients to enroll in PHRs but provide increasing benefit to their doing so (North et al., 2011). 

Such tools can be designed to help answer two of the most fundamental questions patients can ask: 
1) For my condition, which treatments work better than others? and 2) Which treatments are most 
likely to work best for me? Comparative Effectiveness Research (CER) can help answer the first question 
and provide a model for addressing the second. The Congressional Budget Office has defined CER as 
"...a rigorous evaluation of the impact of different options that are available for treating a given medical 
condition for a particular set of patients" (CBO, 2009). The Institute of Medicine (2009) goes further: 

CER is "the generation and synthesis of evidence that compares the benefits and harms of alternative 
methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. 
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The purpose ofCER is to assist consumers, clinicians, purchasers, and policy makers to make informed 
decisions that will improve health care at both the individual and population levels." CER is a feature of 
the American Recovery and Reinvestment Act of 2009 (Iglehart, 2009) and the Patient Protection and 
Affordable Care Act of 2010. It is highly integrated into the US Department of Veterans Affairs (VA) and 
other health-related federal agencies (O'Leary et al., 2010). Although Mushlin and Ghomrawi (2010) 
assert the strategic importance of CER as an evidence-based form of cost containment, it has not fully 
evolved to provide efficient methodologies which can be used on the frontlines of care to control costs 
(Schneeweiss, 2007) and is still in relative infancy for use directly with frontline providers and patients. 

A growing body of personalized medicine research has developed the concepts of individualized 
comparative effectiveness (Epstein and Teagarden, 2010; Thomas et al., 2010; Hoffman and Podgurski, 
2011) and outcomes-based treatment decisions (Crilly, 2012; Esmaily et al., 2010). Studies have drawn 
upon enterprise clinical data warehouses and population databases (Alemi et al., 2011) to evaluate the 
composite treatment experiences for specific medications and to develop statistical methods to identify 
and evaluate these data (Alemi et al., 2009). Tapping directly into the EHR itself is the next logical step. 
EHR systems form the important information core that contains individual patient-level treatment and 
outcome data. Large EHR systems such as the US Department of Veterans Affairs' (VA) VistA (Veterans 
Health Information and Technology Architecture, Brown et al., 2003) and Epic ( www.epic.com ) have 
made substantial headway in maximizing actionable data elements and eliminating non-searchable text 
fields. Several studies drawing upon EHR data in this manner have shown positive results. For example, 
a study by Pakhomov et al. (2008) used a "bag-of-concepts" method to extract quality of life data from 
the text of physicians' EHR notes. They concluded that physicians' notes can be used to calculate a 
surrogate quality of life rating for pain. Tannen et al. (2009) also found that methods to decrease the 
effects of confounding variables allows EHR data to more accurately measure therapeutic effectiveness. 

The increasing availability of data granularity allows the building of targeted, patient-specific 
applications (Ho et al., 2012) to positively impact not only on delivery of care, but patient's choice of 
treatments. With the Patient Protection and Affordable Care Act already having an effect on the 
construction, content, and rapid proliferation of electronic health record systems (Elnahal et al., 2011), 
there are abundant opportunities to increase the value of PHR and EHR systems both to healthcare 
providers and consumers. 

Methods 

Project Description : 

Using such approaches as guidelines, we built an "Outcomes-Based Prescribing" (VA-OBP) system to 
inform the normal prescribing process of clinicians and patients by making use of aggregate EHR data. 
Typically, patient outcomes are evaluated prospectively after the start of a medication and doses are 
changed accordingly. We were interested in developing methods to determine approximate or 
expected outcomes on a given treatment for a given patient based on how other patients similar to 
them have done on those same medications and doses. We sought to get as close to approximating 
potential outcomes for an individual patient as possible by building a comparison cohort of patients 
sufficiently similar to that individual. The output would provide the clinician and patient a compilation 
of outcomes and a summary of the range of experience that similar patients have had with a particular 
medication. This paper describes the conceptual model of the system, the system architecture, and 
presents screen impressions. We then report our conclusions and outline next steps. The purpose of 
this project was to develop proof-of-concept and software operationalization of the Outcomes-Based 
Prescribing conceptual model. The project was funded by the Greenfield Innovations Program of the US 
Department of Veterans Affairs. 
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Conceptual Model: Venue of Use: 

Patient-provider interactions are complex, but when providing new or different treatment, can be 
outlined across 4 steps: 

1. Patient receives a new diagnosis or needs to start on a new medication 

2. Patient reviews information on disease, identifies and becomes familiar with potential 
treatments 

3. Patient discusses treatment options with provider 

4. Patient and provider jointly reach treatment decision 

Adding OBP will provide two additional steps, occurring between steps 3 and 4 above: 

1. Patient and provider view data on how other patients like themselves have fared on the 
different treatment options under consideration 

2. Patient and provider view additional data on cost, generic availability, FDA warnings, and FDA 
adverse events 

Conceptual Model: Software Development/Design Tasks: 

We identified four key development categories: 

1. To develop processes for data retrieval and definition to allow identification of the index patient 
and then the de-identified comparison group. 

2. Draw descriptive information about the medication from outside sources, such as when the 
medication received FDA approval, how long it has been on the market, the length of time 
remaining on the patent, and whether generic versions exist for off-patent dosage units. 

3. To draw outcomes data for the comparison cohort from the EFIR. These data are used to 
characterize the outcomes of the target medication for the reference patient based on the real-time 
outcomes and experiences of similar patients. Data will include physical, mental, and social 
functioning fields. 

4. To calculate, analyze, and report outcomes back to the provider in a useful format. Calculation and 
analysis will include integrating the data to produce easy-to-understand results which are relevant 
to the needs of the prescriber and patient. 


System Architecture: 

The VA-OBP system was designed by the lead 
author (JFC) and built by ICF (WAB) under a VA contract 
with CWI. Data and medical parameters were 
determined by authors JFC and LCTL. VA-OBP is a multi¬ 
layered Java J2EE application built to run on a web 
application server. Data are presented to the user via a 
web front-end. All system data are stored on an Oracle 
XE database. The application can be run on any J2EE 
application server and contains the following 
components: 

Presentation Laver - JSF 2.0 : The presentation layer uses 
JSF 2.0 to display data to the user via a web browser. JSF 
enables AJAX functionality resulting in partial form 
submits which produces immediate webpage updates 
without refresh. JSF components also provide rich 
interfaces like tabbed panels. 
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Persistence Laver - Hibernate 3.2.5 : The persistence layer is an abstract layer that separates direct 
database calls from business logic. Java objects are mapped to database tables via Hibernate 
configuration files. Once mapped, these persistent objects allow access to database data through their 
various method calls. 

Data Injection - Spring 2.5.6 : The presentation layer accesses its display data through calls to backing 
beans. Backing beans use data access objects (DAO) to manipulate Hibernate persistence objects. The 
DAOs are initialized with Spring. 

Business Logic Layer : Data pull rules and parameter dictionary, created separately and updated based 
on new research findings, best practices, and revised clinical guidelines. 

Build - Maven 2.2 : VA-OBP source code was built using Maven. A system war file was created by 
running the install goal of the Maven pom.xml file. Because Maven will download all necessary jar files, 
these dependency files do not need to be locally available when building. Maven gathers these files and 
places them in the repository if they are not already there. 

Data Model 

We sought to formulate a dynamic dataset of EHR and other auxiliary data that can be frequently 
updated. Although we initially sought to use "live" VistA data, running VA-OBP would cause a 
substantial drag on live system operation. Instead, we designed structured query call data from the 
local VistA system, creating a relatively small dataset for OBP use. Theoretically, VA-OBP dataset 
updates can be run overnight or on weekends and, depending on the number of medical specialty areas 
OBP draws from, may produce a substantial stand-alone set. This can be designed and managed 
different ways depending on the individual application. For this project, we used only VistA CPT code 
data fields that were codified and commonly available. To choose the fields to use, a small expert 
committee examined specific data fields and identified potential choices based on data field relevance, 
availability, completeness, and clarity of data definition. We then analyzed these data fields in large 
patient extracts using SAS (ver. 9.2) to determine fields with sufficient data to support OBP. 

Although we examined actual patient data for determining fields to include in the model, we used 
only simulated data in building the prototype. Data field categories included patient demographic, 
diagnosis, service utilization (CPT, Visit, Appointment), treatment (medication-related), 
assessment/diagnostic (Vitals), and mental health (from the VA Mental Health Assessment battery). We 
also tested the ability to integrate useful information from other sources. We created an Excel dataset 
of medication information including patent and manufacturer information manually extracted data from 
the FDA Orange Book ( www.accessdata.fda.gov/scripts/cder/ob/defautlt.cfm) . This information is not 
included in pharmacy or EHR databases but can be useful in the prescribing process. The data map is 
presented in Figure 2. 
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Figure 2 

VA-OBP Data Map 



Data Parameters: 

We performed literature searches, identified evidence-based practices, and sought expert 
opinion to build data search parameters for the following areas: medications, reference group elements 
(patient descriptive information), outcomes elements (social, services utilization, clinical/symptoms, 
satisfaction, physical, and mental). For medication, we offered only two class choices: Antidepressants 
and Antipsychotics. All medication names for each class in the VA formulary were entered. 

Reference Group Build Elements 


Table 1 

Category and Description of Data Parameters 


Category 

Description 

Patient Descriptive Information 

Gender 

No parameters necessary beyond M/F designation of the index patient 

Age 

+/- 2 years of the age of the index patient (regardless of decade born) 

BMI 

+/- 2% of BMI 

Race/Ethnicity 

Categorized into White, Black, Hispanic, Native American, Asian, Other 

Mental Health 

Diagnosis 

Diagnoses occurring separately or co-occurring: specified, not inferred. 

• Psychotic Disorder or Diagnoses "with Psychotic Features" 

• Depressive Disorders 

Outcomes Elements: [Baseline=at start of index meds; End=stop of meds OR to today's date) 

Social 

Employment Status and Homeless Status: 

• Change from baseline: Employment or Homeless NO to YES after start of 
meds (i.e., while taking identified medication) OR 

• Change from Baseline: Employment or Homeless YES to NO after meds 

Services Utilization 

Change in pattern of services use (more, less, different pattern, etc.): 


CCSI-NYU: Decision Support Tools and Predictive Analytics in Human Services. 43 














































































































































• Frequency and type of visit (e.g., primary care, mental health) 

• ED vs. Outpatient vs. Maintenance therapy vs. Inpatient 

• Type of provider of service: MD, MH therapist. Primary Care or mental 
health worker. Nurse, OT, voc. rehab., substance abuse counselor, etc. 

Clinical/Symptoms 

• Suicide ideation (increase, decrease, occurrence, re-occurrence) 

• Suicide attempts (incidence) 

• Change in symptoms of depression, anxiety, psychosis 

Satisfaction 

• Drug refilled as scheduled 

• % scheduled appointments kept (all health appointments: primary care, 
medical, maintenance, mental health, etc.). Rate +/- 5% is notable. 

Physical 

• Critical labs: +/-5% change in WBC and neutrophils 

• Weight changes: +/-5% of body mass 

o Meets clinical criteria for obesity (Height/Weight) 

• Blood Pressure: +/-3% change in blood pressure (regardless of WNL) 

Mental 

• Suicide-related behavior: Emergence OR decrease OR none 

o Self-injury 
o Suicide attempts 
o Calls to Hotline 

• PTSD symptoms (Increase/Decrease) 

• Depression symptoms (Increase/Decrease) 

o By 1 or more measurement points from Baseline 
o No return to Baseline if symptom increase after 6 weeks 


Results 


Although not evaluated by users outside of the development team, our goal was to make an 
operational system that is as simple as possible. We sought to include in the design only a minimum 
number of screens with an uncluttered presentation that could be viewed individually or together by 
the prescribing clinician or the patient during the same office visit. Figure 3 presents the six screens 
which are the front-end of the VA-OBP system. 

The first screen requests a Log-On procedure to verify the user. This is necessary for a stand¬ 
alone module but can be modified to use in EHR systems with a single sign-on. Moving from left to 
right, next is the Home Page which contains introductory and explanatory statements about OBP. For 
this prototype the system asks for a medical record number (MRN) but can be any case identifier. The 
response to this screen brings the Patient Look-up Screen which identifies the "index patient" and calls 
their data into the matching algorithm. These include demographic, diagnostic, and clinical data to 
define the parameters of the comparison group. The fourth screen (bottom left) requests the user to 
choose the medication of interest. For this prototype, we built in only anti-depressant and antipsychotic 
medication choices. The fifth screen, the Reference Group Output Screen, allows the prescriber and 
patient to see the information on the construction of the Reference Group. Finally, the Outcomes 
Summary Screen presents the Outcomes sought by the users. For this prototype, outcomes data are 
presented in a text format. Background algorithms for this prototype were based on fairly contained 
approaches, primarily fitting patients and their outcomes within or outside of parameters set by the 
investigators. 
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Discussion 

This OBP tool is designed to provide patients and providers CER-type information using up-to- 
date patient outcome information from EHR system data, ideally delivered through the PHR. Since not 
all EHRs are equally robust, it is important that tools such as OBP are developed to use, at minimum, 
common data elements from any EHR system. In the broader context, OBP is designed to be a formal 
patient decision-making tool for personalized medicine. To our knowledge, it is one of only a few 
attempts to design a working individualized CER tool for patients which goes beyond supplying disease 
or wellness information or general patient self-reports to the individual patient situation. Other such 
attempts are the subject of patent applications by Alemi (see patents 7,702,526 - 2010 and 8,145,583 - 
2012). These patents describe processes which seek to primarily predict the outcome of a specific 
medication (citolopram) in specific individuals using genomic and other biological data. OBP is different 
in that it seeks to use more universally available data from EHR systems to help patients and providers 
make choices. OBP can also allow a means by which providers can increase the individuality of their 
treatment by working with patients to help better identify potential initial or next-step treatments. 
Although designed around psychiatric disorders, OBP could theoretically apply to other specialized 
diagnosis categories such as diabetes, spinal cord pain treatment, and interventions around suicide 
prevention activities, areas highly relevant to Veteran populations. 

It is important to note that OBP and individualized comparative effectiveness are predicated on 
the assumption that experience in a group can be predictive of individual outcomes. This is not a new 
concept and can be a core building block of personalized medicine. The rationale is that medical 
discovery has been built on the outcomes of clinical trials using the relatively small purpose-built cohorts 
to show effectiveness of treatments. Outcomes are then applied to large groups of individuals under 
the assumption that the value can be translated. Actual usefulness to patients and providers at the 
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individual level is lost. However, the broad use of patient self-reported outcome sites such as 
www.patientslikeme.org suggests that there is a desire for more individually-based information among 
patients, an acceptance of this style of reporting, and value placed in this kind of feedback. This type of 
information along with Comparative Effectiveness Research have the potential to be major parts of 
healthcare reform over the next decade, propelled by the increased quality, quantity, granularity, and 
accessibility of EHR data. The focus of the optimization of these data must be usefulness and 
applicability. Although CER currently supports policy changes on a large scale, it is also an excellent 
model for tools scalable for the frontline of care. There is always concern about "clutter," when 
clinicians are provided with so many decision-support tools that there is risk that none, or only a few, 
will be used. We were sensitive to the potential feelings of clinicians that tools are designed to more or 
less automate (and thus diminish) their decision-making role in care. The novelty of our approach is its 
focus on the patient and on patient-clinician interaction as our target and integrating it into the EHR and 
PHR. This is different from conventional decision-support designs which integrate both CER and non- 
CER information and data primarily for clinician use. 

Design Challenges 

VA-OBP is not the same as predicting whether a specific medication will work in a specific 
individual, or for tailoring medications to patient's characteristics as found in Alemi's work and patent 
applications. That work relies on having access to highly detailed genomic and physiologic data. These 
data are not widely available, and are not accessible for most patients in most EHRs, so cannot feasibly 
be a part of a practical system at this point (Denney, 2011). Still, there remain a number of design 
challenges which must be addressed to move this into a more practical application. The three primary 
challenges are 1) the development of accurate outcome parameters for comparison groups (e.g., which 
changes in condition can be considered positive, negative, or null?), 2) data access (gaining access to 
EHR data sources), and 3) analysis of data (application of appropriate statistical methods). It will be 
important to develop plans to address these issues before VA-OBP can be used as an applied approach. 

Future Development 

VA-OBP has the potential to operate within a number of functions, such as a stand-alone 
module, as an element integrated into an EHR, and as an element of patient portals or PHRs. Its design 
is novel because it goes beyond the use of the individual's own data and makes use of the power of 
numbers which exist in healthcare systems' EHRs. As a function of the PHR, it can add value, thereby 
increasing its use. 

We intend to pursue 4 primary directions for future development. The first direction would be 
to expand VA-OBP to test its robustness with different sizes of datasets to simulate different sizes of 
healthcare systems. The second direction is to develop a mobile application (or app) which would allow 
greater flexibility and availability to patients and clinicians. Third, we will continue to develop the power 
of our algorithms. These could include a form of decision support to help in the prescription process to 
optimize chances of choosing the right medication first ("prescription optimization"). Fourth, we are 
interested in expanding the patient functionality to add to it the current functionality of patient 
commentary and feedback sites but with the ability to drill down so that patients can really get a better 
sense of the treatment experiences of patients truly like themselves. This will result in expanding the 
availability of CER concepts to many more settings and thus positively impact treatment, outcomes, and 
patient satisfaction with treatment. 

Conclusion; 

This is an area of opportunity for growth and development which can positively impact patients, 
providers, and healthcare systems. Specifically, clinicians who prescribe treatments, particularly 
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medications, lack access to important clinical outcomes data that could help optimize treatment 
responses in their patients. Furthermore, determining which treatment will provide the most benefit 
with the least problems for a particular patient within a particular community remains a significant 
challenge for any prescriber. 
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APPENDIX B 
Curricula Vitae 


Included are CVs for all staff identified at this point: Drs. Crilly, Cleek, and Mercado and Ashley 
Fuss. 
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April, 2014 
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Rochester, NY 
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Psychiatry and Behavioral Sciences, New Orleans, LA 
Director of Special Programs, Community Empowerment Services 
Agency, New Orleans, LA 
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1978-1982 Social Work, The Catholic University of America, Washington, DC (BA) 

1984-1986 Social Work, Fordham University, New York, NY (MSW) 

1997-2005 Public Health, University of Rochester, Rochester, NY (MPH) 

1997-2005 Health Services Research and Policy, University of Rochester (Ph.D.) 
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1982 

1986 

2005 

2005 


Bachelor of Arts and Science 

MSW, Master of Social Work (NYS Lie.#: 034378-1; Cert.# 6461748) 
MPH, Master of Public Health 

Ph.D., Doctorate of Sciences in Health Services Research and Policy 
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Coordinated Care Services, Inc (CCSI) 

Rochester, NY 

Duties: Create Ql department, oversee data management, data analysis, 
reporting, develop customer base, develop and implement Continuous 
Quality Improvement methods in work with counties, state, and private 
entities. 


3/2013 - 10/2013 Director of Qperations 

New York State Qffice of Mental Health 
Rochester Psychiatric Center 
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Rochester, NY 

Duties: Oversee clinical and strategic operations of all adult inpatient and 
forensic services. 


1/2011 - Present 


1/2011 - 06/2013 


1 / 2011 - 8/2012 


2/2010 - Present 


8/2009-2/2010 


7/2008-7/2009 


7/2007-12/2010 


(Adjunct) Assistant Professor 

Tulane University Medical Center, Dept, of Psychiatry 
New Orleans, LA 

Duties: Research, data management/analyses, subject expertise electronic 
health records, electronic communication technology, mental health, 
health informatics 

Health Sciences Researcher Affiliated Health Sciences Researcher 
VISN 2 Center of Excellence for Suicide Prevention 
Canandaigua VAMC, Canandaigua, NY 

Duties: Research, manuscripts, grant writing, subject expertise: electronic 
health records, electronic communication technology, mental health, 
health informatics 

Health Sciences Researcher 
VISN 16 MIRECC 

Southeast Louisiana Veterans Health Care System (SLVHCS) 

New Orleans, LA 

Director, Special Programs 

Community Empowerment Service Agency (CESA) 

New Orleans, LA 

Duties: Subject expert, program/business development, supervision 

Principal Epic Rx Trainer, Department of Pharmacy 
Epic Electronic Health Record Implementation Team 
Information Systems Division 
University of Rochester Medical Center 
Rochester, NY 

Duties: Subject expert, training, EHR programming 

Certification Commission for Healthcare Information Technology 
(CCHIT): Member, Behavioral Health Work Group 
Duties: Expert advisement 

Health Sciences Researcher and Data Systems Manager 
Center of Excellence at Canandaigua 
Veterans Health Affairs, Canandaigua VAMC 
Canandaigua, NY 

Duties: Research, website design, clinical applications 
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2/2005-12/2009 


6/1991-2/2005 

1990-1991 


1989-1990 

1986-1989 

1982-1986 
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1/2011-Present 

3/2010-12/2010 

2/2008-2/2010 


Director, Clinical Outcomes 

Department of Psychiatry 
Core Advisory Team, Electronic Health Record 
Information Systems Division 
University of Rochester Medical Center 
Rochester, NY 

Duties: Research, data management/analyses, EHR consultation 

Treatment Team Leader 
New York State Office of Mental Health 
Rochester Psychiatric Center 
Rochester, NY 

Duties: Management, supervision, treatment development 

Sr. Programmer/Analyst 

University of Rochester Medical Center 

Department of Psychiatry 

And Rochester Psychiatric Center 

Rochester, NY 

Duties: Clinical research, data management/analyses, programming 
Social Worker 

New York State Office of Mental Health 
Willard Psychiatric Center 
Willard, NY 

Duties: Clinical work with patients, quality assurance, data analyses 

Clinical Research Analyst 
New York State Psychiatric Institute 
Columbia University 
New York, NY 

Duties: Data management (SAS/SPSS/BMDP), analyses, training 

Mutual Funds Administrator 
Donaldson, Lufkin & Jenrette 
Alliance Capital Management 
30 Broad Street 
New York, NY 

Duties: Fund management, purchase/sale short-term securities 


Assistant Professor, Tulane University, Department of Psychiatry and 
Neurology, New Orleans, LA (Adjunct since 8/2013) 

Clinical Assistant Professor, University of Rochester, Department of 
Psychiatry, Rochester, NY 

Assistant Professor, University of Rochester, Department of Psychiatry, 
Rochester, NY 
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2/2005-2/2008 


Senior Instructor, University of Rochester, Department of Psychiatry, 
Rochester, NY 


Teaching and Other Faculty Appointments : 

2/2011-11/2011 William Carey University, New Orleans, LA, Adjunct Professor 


2/2007-3/2014 

(Courses taught: Consumer Health, Social Problems, Introduction to 
Social Work) 

University of Rochester, School of Nursing and Warner School of 
Education, Rochester, NY, Ph.D. Advisor 

9/2006-12/2009 

University of Rochester, Department of Community and Preventive 
Medicine, Rochester, NY, Adjunct Instructor (Course taught: 

Introduction to Health Services Research, Doctoral Program) 

5/2006-5/2008 

Greater Rochester Collaborative MSW Program, Nazareth College and 
SUNY Brockport Schools of Social Work, New York, Adjunct Instructor 
(Courses taught: Social Work Practice, Research Methods) 

1/1995-12/2007 

State University of New York at Buffalo, Department of Social Work, 
Buffalo, New York, Clinical Assistant Professor (Courses taught: Social 
Work Practice, Research Methods, Organizational Theory, Personality 
Disorders) 

Dissertation Committees: 

10/2009-3/2014 

University of Rochester Warner School of Education 

Candidate: Khamkay Chitaphong 

Reviewer for: 



Psychiatric Services 

Mental Illness (at www.pagepress.org/mi ) 

Value in Health 

Health Services Research (HSR) 

eHealth and Mental Health QUERIs, US Department of Veterans Affairs (Grant Reviewer) 


Board Membership: 

12/2013 - Present 

6/2005 -10/2012 

Compeer, Inc., Rochester, NY 

Coordinated Care Services, Inc. (CCSI), Rochester, NY 

VI. Professional Affiliations 
Workgroups 

4/2011-8/2013 

MyHealtheVet Personal Health Record Peformance Workgroup. US 
Dept, of Veterans Affairs, VA Central Office 

2/2012 -12/2012 

MyHealtheVet Policy Revision Workgroup. US Dept, of Veterans Affairs, 
VA Central Office. 

1/2013 - 8/2013 

National Executive Committee, VITAL Initiative, US Dept, of Veterans 
Affairs, VA Central Office 

2/2013 - 8/2013 

VA Workgroup for Enterprise Text Messaging, US Dept, of Veterans 
Affairs, VA Central Office 
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Grants 

Community Agency Grants 

Louisiana Workforce Commission, Terrebonne Parish: OJT Contract with Pearl Inc. Workforce Re- 
Training, Gulf Coast Fishing/Seafood Industry: Total Grant: $264,000 
Community Empowerment Agency (CESA) 

02/28/2011-06/30/2012 

Louisiana Workforce Commission, Jefferson Parish: Specialized Training Contract 
Certified Translation Services, LLP: Total Grant: $102,620 
Community Empowerment Agency (CESA) 

04/15/2011-06/30/2012 

Private Investment/Venture Capital 
Special Business Entity: Fin International, LLP 
Total Investments secured to date: $210,000 
Community Empowerment Agency (CESA) 

10/1/2011-Present 

Academic Grants 

US Department of Veterans Flealth Affairs, Center of Excellence for Suicide Prevention: Total Grant: 

$ 100 , 000 . 

"Keeping In Touch" 

03/31/2011 - Present 

PI: Sonya Batten, PhD; Robert Bossarte, PhD 
Site Investigator: John Crilly, PhD 
Status: COMPLETE 

US Department of Veterans Affairs, VISN 16 MIRECC: Total Grant: $20,000 

"Clinical Demonstration Project for Increasing Treatment Engagement with OEF/OIF/OND Veterans. 

10/2011-9/2012 

PI: John Crilly, PhD, MPH, MSW 

Co-PI: Frederic Sautter, PhD 

US Department of Veterans Flealth Affairs, Center of Excellence for Suicide Prevention: Total Grant: 
$100,000 (Coverage: 2/8 FTE). 

"Keeping In Touch" 

03/31/2011 - Present 
PI: Ira Katz, MD, PhD 
Co-PI: Kenneth Conner, PsyD 
Site Investigator: John Crilly, PhD 
Status: COMPLETE 

US Department of Veterans Flealth Affairs, Office of Information and Technology, Greenfield 
Incubation Award for Flealth Information Technology Innovation: Total Grant: $150,000 

"Outcomes-Based Prescribing" 

06/01/2009-5/31/2011 

PI: John Crilly, PhD 

Co-PI: Diane Niemann, VISN2 CIO 
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Status: COMPLETE 


New York State Office of Mental Health: Total Grant: $37,000 

"Community-based Surveillance for Health Outcomes and Service Utilization", 1/01/2008 - 

12/31/2008 

PI: John Crilly, PhD 

Co-PI: Eric Caine, MD 

Status: COMPLETE 

New York State Office of Mental Health: Total Grant: $30,000 

"Review of Acute Services Documentation for Patients at Elevated Risk for Suicide", 1/01/2007 - 

12/31/2007 

PI: John Crilly, PhD 

Co-PI: Eric Caine, MD 

Status: COMPLETE 

Awards 

September, 2012: "Outcomes Based Prescribing" awarded as Top Government IT Innovation by 
InformationWeekSOO 

Peer Reviewed Publications 
In Process : 

Chitaphong K, Crilly JF. Engaging military students: Addressing the challenges of maximizing 
involvement of returning OEF/OIF Veterans in VA higher education benefits. Military Health . 

Crilly, JF, Conner K, Beutrais A, Silenzio V. The Next Generation of Psychological Autopsy 
Studies. Part 3. Electronic Communications. Suicide and Life Threatening Behaviors . 

Submitted, Under Review : 

Crilly JF, Luu LCT. Outcomes-Based Prescribing: A Novel Use of Electronic Health Record 
Data to Inform Patient and Clinician Treatment Choices. British Journal of Medical 
Informatics . 

Crilly JF, Keefe RH, Voipe F, Taylor M. Advancing online behavioral health treatment in non- 
urban/rural areas. Community Mental Health Journal . 

Published : 

Crilly JF (2013). Building Connections: Strategies to Mitigate Suicide Risk in Civilians and Military 
Veterans Living in Non-Urban and Rural Areas. In Frontiers of Suicide Treatment, (J. Lavigne, 
Ed.). 

Crilly JF, Keefe RH, Voipe F (2011). Use of Electronic Technologies to Promote Community and 
Personal Health for Individuals Unconnected to Healthcare Systems. American Journal of Public 
Health, 101(7): 1163-7. 

Britton PC, Bossarte R, Lu N, He H, Currier G, Crilly JF, Richardson T, Tu X, Knox K (2011). 
Prevalence, correlates, and symptom profiles of depression among men with a history of 
military service. Social Psychiatry and Psychiatric Epidemiology, 46(7): 607-14. 
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CrillyJF, Lewis J (2010). Internet-Based Psychiatric Interventions: Applications for Rural 
Veterans at Risk for Suicide. Proceedings of the International Conference on Society and 
Information Technology, April. 

Erickson S, Lamberti S, Nihalani N, Crilly J, Weisman R, Desai R (2009). Predictors of Arrest 
During Forensic Assertive Community Treatment. Psychiatric Services, 60(6): 834-7. 

Crilly JF, Caine ED, Lamberti SJ, Brown T, Friedman B (2009). Mental Flealth Services Utilization 
and Symptom Prevalence in an Adult Probation Cohort. Psychiatric Services, 60(4): 542-4. 

Crilly JF. (2008). An Overview of Compulsory, Non-Compulsory, and Coercive Interventions for 
Treating People with Mental Disorders in the United States. International Journal of Mental 
Health, 37(3): 57-80. 

Wang D, Crilly JF, Jaeger LA, Palmer G (2007). Assessing Patient Preferences for Reminder 
Delivery in a Psychiatry Ambulatory Service. Proceedings of the American Medical Informatics 
Association, November. 

Crilly JF. (2007): The history of clozapine and its emergence in the US market: a review and 
analysis. History of Psychiatry, 18(1): 39-60. 

Lamberti JS, Olson D, Crilly JF, Olivares T, Williams GC, Tu X, Tang W, Wiener K, Dvorin S, Dietz 
MB. (2006). Prevalence of the metabolic syndrome among patients receiving 
clozapine. American Journal of Psychiatry, 163(7): 1273-6. 

Lamberti JS, Costea GO, Olson D, Crilly JF, Maharaj K, Tu, X, Groman A, Dietz MB, Bushey 
MP, Olivares T, Wiener K (2005). Diabetes mellitus among outpatients receiving 
clozapine: prevalence and clinical-demographic correlates. Journal of Clinical Psychiatry, 
66(7): 900-6. 

Crilly, JF. Affecting Change in the Use of Restraint and Seclusion in a State Hospital 
Setting. University of Rochester Medical School, Masters Thesis: May, 2005. 

Crilly, JF. Mental Health Services Utilization of People with Symptoms of Mental Illness With 
And Without Involvement in the Criminal Justice System: Correlates and Estimators to Help 
Guide Policy and Service Development. University of Rochester Medical School, Doctoral 
Dissertation: March, 2005. 

Crilly JF, Cain N, Davidson P. "Risk factors for mood disorders in people with intellectual 
disability", in Mood Disorders in People with Mental Retardation, Peter Sturmey, Ph.D, Editor. 
2005. 

Lamberti JS, Crilly JF, Maharaj K, Olson D, Wiener K, Dvorin S, Costea GO, Bushey MP, 

Dietz MB. (2004). Prevalence of diabetes mellitus among outpatients with severe mental 
disorders receiving atypical antipsychotic drugs. Journal of Clinical Psychiatry, 65(5): 702- 
706. 
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Schwarzkopf SB, CrillyJF, Silverstein SM. (1999) Therapeutic synergism: optimal 
pharmacotherapy and psychiatric rehabilitation to enhance functional outcome in 
schizophrenia. Psychiatric Rehabilitation Skills, 3(1), 124-147. 

McFarlane WR, Link B, Dushay R, Marchal J, Crilly JF. (1995) Psychoeducational Multiple family 
groups: four-year relapse outcome in schizophrenia. Family Process, 34(2), 127-144. 

Lambert! JS, Schwarzkopf SB, Boutros N, Crilly JF, Martin R. (1993) Within-session changes in 
sensory gating assessed by P50 evoked potentials in normal controls. Progress in 
Neuropsychopharmacology and Biological Psychiatry, 17(5), 781-791. 

Bradt S, Crilly JF, Timvik U. (1993) Computer training for the young adult patient with chronic 
mental illness". Journal of Rehabilitation, July, 51-54. 

Kane CF, Testani M, Crilly JF, Auberger K, Norton B. (1992) Predicting the need for 
long-term hospitalization of severely mentally ill young adults. Journal of Flospital and 
Community Psychiatry, 43(12), 1239-1241. 

Schwarzkopf SB, Lambert! JS, Crilly JF, Martin B, Flirt J, Flolley L. (1991). Combined startle and 
P50 measures of sensory gating: support for a shared neurophysiology. Society of Biological 
Psychiatry . 

Crilly JF. (1986) Analysis of Characteristic Differences and Similarities Between Floused and 
Flomeless Public Assistance Families. Unpublished Masters Thesis, Fordham University Graduate 
School of Social Service, May, 1986 

Academic Presentations 

Oral and Poster Presentations 

Crilly JF (2012) A Novel Use of Electronic Flealth Records Data to Inform Clinician and Patient 
Treatment Choices . VIREC online presentation: March 22. 

Crilly JF, ConstansJ, Sullivan G (2010). "Flealth-Related Internet Use by Rural Residents in 
Veterans Integrated Service Network (VISN) 16 and Central South U.S." American Public Flealth 
Association Conference, November, Oral Presentation. 

Crilly JF, Claassen C, Sullivan G (2010). "Internet Use for Flealth-Related Needs: Implications for 
Developing Online Mental Flealth Interventions for Veterans." American Public Flealth 
Association Conference, November, Poster Presentation. 

Crilly JF (2010). "Potential for Delivering Online Suicide Prevention to At-Risk Veterans." VA 
Mental Flealth Conference, July 27. 

Chitaphong K, Crilly JF (2010). "Can we Predict Use of Education and Job Training Benefits by 
Returning OEF/OIF Veterans?" VA Mental Flealth Conference. Baltimore, July 28. 

Crilly JF (2010). "Delivering Electronic-Based Mental Flealth Services: Factors, Considerations, 
and Cautions." Grand Rounds, University of Arkansas, Department of Psychiatry, June 3. 
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Crilly JF (2010). "Health-Related Internet Use by Residents of VISN 3 and VISN 16" at the 2010 
National Meeting on Improving Health Care for Rural Veterans. Poster Presentation. 

Crilly JF (2010). "Internet-Based Psychiatric Interventions: Applications for Rural Veterans at 
Risk for Suicide" at the April, 2010 International Conference on Society and Information 
Technologies. Paper Presentation. 

Crilly JF (2010). "Therapeutic Use of Coercive Measures." Grand Rounds, Tulane University 
Department of Psychiatry, January 8. 

Crilly JF (2009) "Sinking, Swimming, or Just Splashing Around: Facing Motivational Challenges 
as a Middle Manager." December 16 and 17, VISN 2 Supervisor Live Meeting, Invited Speaker. 
Oral Presentation. 

Crilly JF, Lewis J (2009) "Presence of Suicide Prevention Content in Military or Veteran-Related 
Blogs and Forums" at the November, 2009 American Public Health Association (APHA) Annual 
Meeting. Oral Presentation. 

Crilly JF (2009) "Basic Research Process in the VA: Evidence-Based Practice", May, 2009 at 
Canandaigua VA Medical Center. Oral Presentation. 

Luu LC, Crilly JF, Newcomer B, Smith S, Cefalu W (2009) "Intermuscular Adipose Tissue in Soleus 
Muscle is Increased in Type 2 Diabetes" at the June, 2009 American Diabetic Association (ADA) 
Annual Meeting. Poster Presentation. 

Crilly JF, Bossarte R, Britton P (2009) "Age differences in Psychological Distress and Mental 
Health Services Use Among Veterans" at the March, 2009 Behavioral Risk Factor Surveillance 
System (BRFSS) Annual Meeting. Oral Presentation. 

Crilly JF (2008). "Alternative Methods for Accessing Individuals Unconnected with the 
Health Care System: Developing Extant Non-Health Community-based Data for Health 
Services Intervention Planning". Expert Level Session: 18**^ Annual Summer Institute in 
Nursing Informatics, University of Maryland: Baltimore, MD, July 17, 2008. 

Crilly JF, White AM, Keefe R (2007). "An Alternative Model to Bridge Urban Health Disparities 
and RHIO Benefits". 6*^ Annual International Conference on Urban Health: Baltimore, MD, 
October 31, 2007. Oral Presentation 

Wang D, Crilly JF, Jaeger LA, Palmer G (2007). "Assessing Patient Preferences for Reminder 
Delivery in a Psychiatry Ambulatory Service." American Medical Informatics Association 
Conference: November 10, 2007. Oral Presentation. 

Crilly JF (2007) "Future and Outcomes: VA Knowledge and Information Week Lecture": Lecture 
for the Canandaigua, NY Veterans Affairs, March 29. Oral Presentation. 

Crilly JF, Wang D, Jaeger LA, Palmer G (2007). "Client Preferences for Reminder Delivery 
Methods in a Psychiatry Ambulatory Service." Collier Symposium and Presentation, March, 
2007. Poster Presentation. 
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Crilly JF (2006) "The history of clozapine and its emergence in the US market": Lecture for the 
Corner Society of the History of Medicine, November 29. Oral Presentation. 

Crilly JF, Aldige-Hiday V, Broner N, Currier G. "Are Pre-therapeutic Engagement Techniques in 
Restrictive Settings Coercive?" American Public Health Association Conference: Boston, MA, 
November, 2006. Oral Presentation. 

Crilly JF, Cerulli C, Hall D, Campbell A (2006). Case comparison of one-year mental health court 
outcomes: unfunded vs. funded courts. American Society of Criminology Conference: Los 
Angeles, CA, November, 2006. Poster Presentation. 

Crilly JF, Cerulli C, Hall D, Campbell A (2006). Monroe County Mental Health Court Outcome 
Evaluation. Collier Symposium and Presentation, March, 2006. Poster Presentation. 

Crilly JF (2006) "The history of clozapine and its emergence in the US market": Grand Rounds, 
Rochester Psychiatric Center, February 9. Oral Presentation. 

Crilly JF (2005) Organized symposium "Evidence-based Practice for Mental Health Services 
Consumers with Criminal Justice System Involvement" at the December, 2005 American Public 
Health Association (APHA) Annual Meeting. Oral Presentation. 

Crilly JF (2005) "Overview of Research on Service Use by People with Mental Disorders 
Involved in the Criminal Justice System", Symposium, December, 2005 American Public Health 
Association (APHA) Annual Meeting: Oral Presentation. 

Luu LCT, Crilly JF, Kasarskis, EJ. (2004) A self-report scale measuring denial in ALS 
patients andtheir caregivers. December, 2004, Fifteenth International Symposium on ALS and 
Motor Neuron Diseases, Philadelphia, PA. Poster Presentation. 

Crilly JF, Friedman B, Lamberti JS. (2004). Mental health services utilization by mentally ill 
offenders on probation. November, 2004, American Public Health Association Conference, 
Washington, DC. Poster Presentation. 

Schwarzkopf SB, Badgely J, Light G, Fugle T, Guardinino J, Williams Carol, Crilly JF. 

(2003).State-linked quantitative measures of symptomatology, cognition, and brain electrical 
activity in the clinical treatment of psychosis. April, 2003, IXth International Congress on 
Schizophrenia Research, Colorado Springs, CO. Poster Presentation. 

Schwarzkopf SB, Badgely J, Light G, Fugle T, Guardinino J, Williams Carol, Crilly JF. 

(2002). Integration of state-linked qualitative measures of symptomatology, cognition, and 
brain electrical activity in the treatment of psychosis. NYS Office of Mental Health Research 
Conference, (December, 2002: Poster). 

Lamberti JS, Crilly JF, Maharaj K, Olson D, Costea O, Bushey P, Quallo H, Dietz M. (2002) 
Prevalence of Diabetes Mellitus Among Outpatients Receiving Antipsychotic Drugs. October, 
2002, Psychiatric Services Conference, Chicago, IL. Poster Presentation. 
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Crilly JF, Burget M, BellnierT. (2001) Freeing Up the "Tie-Down" Mentality: Adjusting 
Staff Attitudes Away From Reliance on Restraint and Seclusion. December, 2001 NYS Office of 
Mental Flealth Research Conference, Albany, NY. Poster Presentation. 

Crilly JF, Luu LC. (2001) Qualitative Assessment of Informants' Experience of Mental 
Illness, Intensive Flospitalization, and Transition Back to Their Community. October, 2001, 
American Public Flealth Association Annual Meeting, Atlanta, GA. Poster Presentation. 

Schwarzkopf SB, Bowman J, Davenport D, Pierce D, Crilly JF, Porosoff G. (2000). 

"Behavior Tracking Initiative: A Performance Improvement Project from Conceptualization to 
Implementation" December, 2000, NYS Office of Mental Flealth Research Conference, Albany, 
NY. Poster Presentation. 

Crilly JF, Luu LC, Stapholz B, Smith A. (2000) "Maintaining the Family Unit for Mothers and 
Young Children" December, 2000, NYS Office of Mental Flealth Research 
Conference, Albany, NY. Oral Presentation. 

Crilly JF. (2000) "Use of a Specialized Inpatient Psychiatric Setting to Divert Clients from 
Enrollment in a Mandated Outpatient Treatment Program" June 2000, National Research 
Service Awards (NRSA) Conference, Los Angeles, CA. Oral Presentation. 

Ocon R, DiMartino ED, Crilly JF. (1998) "Team-Building in Clients With Mental Illness" 

NYS Office of Mental Flealth Research Conference (December. 1998: Poster). 

Crilly JF, DiMartino ED, Mathew T. (1998) "Insight Into Illness: A Detriment or an 
Enhancement?" Workshop at the 1998 American Psychiatric Association Conference in Toronto, 
Canada, (June, 1998: Presentation) 

Crilly JF, D'Anza WF, McCarthy S. (1997) "Characterization of Long-Term Patients in a 
State Flospital Setting". NYS Office of Mental Flealth Research Conference (December, 1997: 
Poster). 

Crilly JF, Herz ML (September, 1997) "An intermediate length-of-stay treatment program: 
Extension of the acute treatment setting for 'revolving door' clients." Invited presentation at 
"Psychiatric Rehabilitation of Schizophrenia: Current Trends and Future Directions" conference 
in Rochester, N.Y. 

Crilly JF, Jackson C, Martlock C, Cunico L. (1996) "Impact of a non-smoking policy on 
two inpatient programs at a state psychiatric center." NYS Office of Mental Flealth Research 
Conference (December, 1996: Poster). 

Crilly JF, Andrieu C, Jackson C. (1996) "Community success: the impact of prior work 
history in conjunction with active participation in vocational programs on an intermediate 
treatment program." NYS Office of Mental Flealth Research Conference (December. 1996: 
Poster). 
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Crilly JF, (1996, 1998) Keynote speaker for the Committee to Aid Research to End 
Schizophrenia (C.A.R.E.S.) annual fund-raising dinner (October 4 , 1996 and October 9 , 1998) and 
the Rochester Alliance for the Mentally III (A.M.I.) annual meeting (October 5, 1996) on "Current 
and Future Research in Mental Illnesses". 

Crilly JF, Herz Ml. (1995) "Alternative Inpatient Treatment Setting: 2 Year Follow-up of an 
Intermediate Length-of-Stay Treatment Program". NYS Office of Mental Flealth Research 
Conference (December, 1995). 

Crilly JF, Flerz Ml. (1994) "Integrating MICA treatment within an inpatient psychiatric 
rehabilitation program". NYS Office of Mental Flealth Research Conference (December, 1994: 
Poster). 

Crilly JF, DiMartino E, Flerz Ml. (1994) "Addressing denial of mental illness within an 
inpatient psychiatric rehabilitation treatment framework". NYS Office of Mental Flealth 
Research Conference (December. 1994: Poster). 

Crilly JF, Flerz Ml, Schneider EJ. (1993) "An intermediate length of stay treatment 
program: Bridging the gap between acute and long-term care". NYS Office of Mental Flealth 
Research Conference (December. 1993: Poster). 

Flerz Ml, Lambert! JS, Schwarzkopf SB, Crilly JF. (1992) Prodromal symptoms in 
schizophrenia: a prospective study and review for the International Congress of Schizophrenia 
Research (April. 1993: Poster). 

Lambert! JS, Schwarzkopf SB, Crilly JF, Martin R. (1993) Sensory Gating Assessed by 

P50 Suppression in Normal Controls: Effects of Changing Stimulus Intensity for the International 

Congress of Schizophrenia Research (April, 1993: Poster). 

Schwarzkopf SB, Smith DA, Lambert! SJ, Crilly JF, Flirt J, Martin R. (1992) P50 
AEP Correlates of Persecutory Ideation and Affective State in Controls: Consistencies with 
Findings In Clinical Populations for the International Congress of Schizophrenia Research (April, 
1993: Presentation). 

Schwarzkopf SB, Lambert! JS, Crilly JF, Martin R, Mitra T. (1991). "Sensory gating 
assessed by P50 Evoked Potentials in controls: methodological issues". Presented as a poster 
session at the American College of Neuropsychopharmacology Conference, (December, 1991: 
Poster) 

Lambert! JS, Schwarzkopf SB, Timvik U, CrillyJF, Flenrichs MFI. (1990) "Weight gain in clozapine- 
treated schizophrenics". Presented as a poster session at the Schizophrenia Research 
Conference, (June, 1990: Poster). 

Crilly, JF, Radio Talk Show Guest Speaker: The Talk Show With Paul Baker (WXXI AM), 

June 5,1991. Topic: Schizophrenia 


CCSI-NYU: Decision Support Tools and Predictive Analytics in Fluman Services. 62 




Andrew F. Cleek, Psy.D. 

NYU McSilver Institute for Poverty, Policy, and Research 
New York University 
41 East 11th Street, 7th floor 
New York, NY 10003 
Ph: (212) 998-9093 
Email: andrew.cleek@nvu.edu 


Highlights 


• 2012: Appointed a Research Assistant Professor, NYU School of Medicine 

• 2012: Joined NYU McSilver Institute for Poverty, Policy, and Research as Senior Research 
Scientist, Executive Officer McSilver-UIBH 

• 2012: Led assumption of UIBH into NYU’s McSilver Institute for Poverty, Policy, and 
Research 

• 2012: Grew UIBH from a $90,000 Operating Budget at its founding in 2004 to 1 mi llion 
dollars of revenue in Eiscal Year 2011-2012 

• 2011: Named Interim Executive Director Coordinated Behavioral Care 

• 2010: UIBH became a partner in Children’s Technical Assistance Center: OMH funded 
initiative to provide technical assistance to all 350 children’s mental health clinics in NYS 

• 2010: Partnered with Health and Hospital Corporation on HEAL 17 project to connect 7 
UIBH agencies to Interboro RHIO 

• 2010: SAMHSA Science to Service Award for Wellness Self-Management (awarded to 
Center for Practice Innovation at Columbia University) 

• 2010: Eaculty: Center for Collaborative Inner-City Child Mental Health Services Research: Mt. 
Sinai School of Medicine 

• 2008: Named Interim Executive Director of Integrated Wellness Partners 

• 2007: Served as Director on Diabetes Co-Morbidity Initiative: implemented diabetes self¬ 
management materials in 19 NYC based providers 

• 2006: Co-Authored (with Salerno, Margolies) Wellness Self- Management Workbook; now 
being used in lOO-i- New York Agencies 

• 2004: Pounding Director of the Urban Institute for Behavioral Health 

Academic Appointments 


7/12-present Senior Research Scientist 

McSilver Institute for Poverty Policy & Research 

1/13- present Research Assistant Professor 

NYU School of Medicine 


1/13-present 


Adjunct Lecturer 

NYU Silver School of Social Work 


1/03-8/06 


Adjunct Professor 
Metropolitan College of New York 
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Professional Experience 


8/04-Present Director 

The Urban Institute for Behavioral Health 


9/02-9/04 


Assistant Psychologist 
Astor Bronx Day Treatment 


8/00-8/02 Assistant Psychologist 

Astor Early Childhood Programs 

8/99-8/00 Psychology Intern 

The Astor Home for Children 

Education 


9/97-12/01 


9/95-8/97 

9/91-6/95 


Spalding University, Louisville. KY 
Psy.D. in Clinical Psychology, 

o APA-approved Program & Internship 
o Graduate Citation for Academic Excellence 

Spalding University, Louisville. KY 
Master’s Degree in Clinical Psychology, 

California State University at Long Beach. Long Beach. CA 
Bachelor’s Degree in Psychology 


Current Grant Awards_ 

8/13-7/15 Mobile Health Solutions for Behavioral Skill Implementation though 

Homework 

National Institute of Mental Health (R34MH100407-01) 

$574,000 

Co-Investigator (PI: McKay) 

1/11-12/15 Clinic Technical Assistance Center 

NYS Office of Mental Health 
$ 5.0 Million 

Co- Investigator (Directors McKay & Hoagwood) 

3/13-5/14 Preparing the Workforce to Serve Homeless Women Veterans & their 

Lamilies 

US Veterans Administration 

$ 120,000 

Co-Director (Director: McKay) 
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1/14- 12/16 


Special Projects Fund: Creating Outcome-driven systems of care 
New York State Health Foundation 
$330,000 

Principal Investigator 

CURRENT CONTRACTS_ 


2011 -Present Coordinated Behavioral Care 

Director 

2013-2014 YMCA 

Director 


2012-Present Long Island Behavioral Alliance 

Director 

2012-Present NAMI Metro 

Director 


PAST GRANT AWARDS _ 

1/11-12/13 HEAL 17 

NYS Department of Health 
Sub to UIBH $600,000 
Sub Contract Director 


2011-2012 Knowledge Empowers Youth 

NYS Office of Mental Health 
$50,000 

Director of UIBH Sub 

2007-2011 Diabetes Co-Morbidity Initiative 

New York State Health Eoundation 

$580,000 

Director 

2008 Implementation of Clubhouse of Suffolk Smoking Cessation Model 

New York City Department of Health and Mental Hygiene 
$30,000 
Director 

2008 Knowledge Empowers You 

New York City Department of Health and Mental Hygiene 

$30,000 

Director 


2007 Smoking Cessation Implementation 

NYS Department of Health 
$18,000 


CCSI-NYU: Decision Support Tools and Predictive Analytics in Human Services. 65 




Director 


2007 Implementation of COPE Behavioral Parent Training 

Integrated Children’s Services 
$15,000 
Director 

2006 Program Development Grant 

United Way 
$15,000 
Director 

2005 Program Development Grant 

United Way 

$15,000 

Director 


OTHER POSITIONS 


• Consultant: National Council for Community Behavioral Health Care (2012-2013) 

• Interim Executive Director: Integrated Wellness Partners (2008-2010) 

• Interim Executive Director: Coordinated Behavioral Care (2011 -2012) 

• Eaculty: Center for Collaborative Inner-City Child Mental Health Services Research: Mt. Sinai 
School of Medicine (2010- 

• Integrated Health Track Co-Leader (2010, 2011) NYAPRS Annual Conference. Kerhonkson, 
NY. 

• Consultant: Mt. Sinai School of Medicine (2008-2011) 

• Committee Member: OASAS Cultural Competence Certification Curriculum Development 
Committee (2006-2007) 

• Steering Committee Member: SAMHSA funded Core Shelter Services for Women with Serious 
Mental Illness Initiative (2005- 2010) 

AWARDS 


2010 SAMHSA Science to Service Award for Wellness Self-Management (awarded to 

Center for Practice Innovation at Columbia University) 


TRAININGS CONDUCTED 


• Wellness Self-Management 

• Knowledge Empowers You (WSM for Transition Aged Youth) 

• Multiple Eamily Groups 

• Diabetes Self-Management 
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PEER REVIEWED PUBLICATIONS 


Vu, C., Mercado, M, Fuss, A., Peri, D., Cleek, A., Cleek, E. (2014). The Business Efficiencies and 
Effectiveness Project: A Learning Collaborative to Improve Business Practices for Mental Health 
Clinics, (in Review) 

Acri, M., Mercado, M., Cleek, A., Dean Asseal, M., McKay, M., Hoagwood, K., Eranco, L. Salerno, 
A. (2014). The Clinic Technical Assistance Center: Building Capacity in New York State’s Child 
Public Mental Health System (in Review) 

Chor, K, Olin, S., Weaver, J., Cleek, A., McKay, M., Hoagwood, K., Horwitz. (2013). 
Characterizing Clinic Adoption in Child Mental Health Initiatives in New York State. (In Press) 

Hoagwood, K., Olin, S., & Cleek, A. (2013). Beyond Context to the Skyline: Thinking in 3D. 
Administration and Policy in Mental Health and Mental Health Services Research, 20: 23-28. 

Jeanie Tse, MD, ERCPC; Elisa Chow, PhD; Rosemarie Sultana-Cordero, MA, LMHC; Marcia Titus- 
Prescott, RN; Ruth Chiles, RD; Andrew Cleek, PsyD; and Elizabeth Cleek, PsyD. (2011). 
Motivation-Based Interventions for Obesity in Serious Mental Illness. Psychiatric Annals 41(10): 
473-477. 

Salerno, A., Margolies, P., Cleek, A., Pollock, M., Gopalan, G., Jackson, C. (2011). Wellness Self- 
Management: An Adaptation of the Illness Management and Recovery Practice 
in New York State, Psychiatric Services 62: 456-458. 

Cleek, A. E. (2002). The Development of the Accord Program: A high school counseling program. 
Dissertation Abstracts International: Section B: The Sciences & Engineering , 62(10-B). [University 
Microfilms] 


Treatment Manuals 

Salerno, A., Cleek A., Euss, A., Beharie, N. (2012). Knowledge Empowers You, 2"'^.. Albany, New 
York, New York State Office of Mental Health. 

Salerno, A., Margolies, P., Cleek A., Wisdom, J., Perez, M. (2009). Knowledge Empowers You: The 
Road to Wellness. Albany, New York, New York State Office of Mental Health. 

Salerno A., Margolies P., Cleek, A. (2006, 2008) Wellness Self-Management Personal Workbook. 
Albany, NY, New York State Office of Mental Health & Urban Institute for Behavioral Health 


PRESENTATIONS 


Cleek, A., Beharie, N., Puss, A. (2013). Knowledge Empowers You: Empowering youth to make 
decisions that work for them. OMH/Children's Mental Health Coalition Conference. Saratoga 
Springs, NY 
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Cleek, A., Mercado, M, Fuss, A. (2013). Using Data in a transformed behavioral health system. 
OMH/Children's Mental Health Coalition Conference. Saratoga Springs, NY 

Cleek, A., Weiscof, G., Lisio C. (2013). Clinics Place in an Evolving Healthcare System. New York 
State Association of Psychiatric Rehabilitation Annual Executive Seminar. Albany, NY. 

McGuirk J., Cleek, A. (2012). Preparing Outpatient Mental Health Clinics for a Challenging Euture- 
Integrating Practice and Business Efficiencies. Preparing for a Changing Primary and Behavioral 
Healthcare System: Cultivating Knowledge & Skills to Meet the Challenge. New York, NY. 

McGuirk J., Cleek, A. (2012). Preparing Children's Outpatient Mental Health Clinics for a 
Challenging Euture. New York State Coalition for Children’s Mental Health Annual Public Policy 
Eorum. Albany, NY 

Cleek, A., McKay, M. (2012). The 4R's and 2S's: Research-informed Practices to Treat Children & 
Strengthen Eamilies. 2012 Pall Behavioral Healthcare Meeting - Changing Tides in the Behavioral 
Health System: Adapt, Evolve, and Thrive. Hamilton, NJ. 

Hoagwood, K., Mckay, M., Salerno, A., Cleek, A. (2012). Sharing Challenges of Child Serving 
Behavioral Health Organizations. 42"“^ Annual National Council Mental Health and Addictions 
Conference, Chicago, IL. 

Cleek, A., Gorges, A. Jarvis, D., Rosenthal, H., Schwartz, E. (2012). The Journey Begins: Prom 
Behavioral Health Centers to Health Homes. 42"“^ Annual National Council Mental Health and 
Addictions Conference, Chicago, IL. 

McKay, M., Cleek, A., Salerno, A. (2012). The 4r’s and 2s’s. Research informed practices to treat 
children and strengthen families. 42"‘^ Annual National Council Mental Health and Addictions 
Conference, Chicago, IL. 

Sheppard, Pojman, Cleek, Hodges, & Haen (2012). Evidence-based group practice with children and 
adolescents. 2012 American Group Psychotherapy Association Conference, New York, NY. 

Cleek, A., Porquer, S., Sheola, R. (2011). Medicaid Transformation: The role of Behavioral Health 
Organizations. ACL Annual Meeting, Saratoga, NY. 

Cleek, A., Cleek, E., Mittleman, M. (2011). Low-cost Technology Driven Solutions to Improve Your 
Practice. 4L‘ Annual National Council Mental Health & Addiction Conference. San Diego, CA. 

Campanelli, P., Chiles, R., Chow, E., Cleek, A., Cleek, E., Lastella, D., Sultana-Cordero, R., Titus- 
Prescott, M., Tse, J., Wofsy, M. (2011). Effectiveness of motivation-based interventions to reduce 
cardiometabolic risk in low-resource psychiatric settings. Scientific and clinical report. Annual 
Meeting of the American Psychiatric Association, Honolulu, HI. 

Cleek, A., Mills, D, Mimnagh, C, Reil, M, Silver, J, Stone, E, Wimmer, J. (2010). Innovations in 
Integrated Physical and Behavioral Health Care. New York State Association of Psychiatric 
Rehabilitation Annual Executive Seminar. Albany, NY. 

Cleek, A., Tse. J., Sultana Cordero, R. (2010). Approaches to Working with Disparate Populations. 
CDC Diabetes Translation Conference 2010. Kansas City, MO. 


CCSI-NYU: Decision Support Tools and Predictive Analytics in Human Services. 68 



Cleek, A., Cruz, C., Downes, B., Forquer, S., Galbreath, Manderscheid, R., Magnabosco, J. (2010) 
Emerging and Senior Leaders Dialogue Panel, ACHMHA Sante Fe Summit. Sante Fe, NM. 

Colonna, D., Margolies, P., Salerno, A., Cleek, A. (2010) Recovery without Fiealth Isn’t Recovery at 
All. The National Council for Community Behavioral Healthcare 40th Annual Conference. Orlando, 
FL. 

Campanelli, P., Cleek, A., Chow, E., Mills, D., Tse, J., LaStella, D., & Sultana-Cordero, R. (2009). 
Pre Conference Institute: Integrating Mental Health and Health Utilizing Evidence Based and Best 
Practices: A Format for Residential and Non-Residential Settings. ACLAIMH 30th Annual 
Conference. Bolting Landing, NY. 

Cleek, A., McKenna, S., Mills, D., Rutter, S. (2009). Integrated Models in Health and Behavioral 
Health Care: The ICE & LEGS CIDP Projects. NYAPRS 27* Annual Conference. Kerhonkson, 
NY. 

Cleek, A., Puff, A., Reil, M., Rutter, S., Silver, J. (2009). Emerging Models of Integrated Health and 
Behavioral Health Care. New York State Association of Psychiatric Rehabilitation Annual 
Executive Seminar. Albany, NY. 

Wale, J., Lewis-Fernandez, R., Cleek, A., Brennan, W. (2008). Health and Mental Health Integration 
for Latinos/as Affected by Mental Illness. Second Annual Latino Mental Health Conference. New 
York, NY. 

Cleek, A., Margolies, P., and Salerno, A. (2008). Wellness Self-Management: What is it? Why is it 
important? How can you use it in your organization? New York State Association of Psychiatric 
Rehabilitation Annual Conference. Ehenvihe, NY. 

Cleek, A., Tse. J., and Campanelli, P. (2008). The treatment of adults with serious mental illness and 
diabetes. The American Psychological Association 116* Annual Convention. Boston, MA. 

Cleek, A., Margolies, P., Salerno, A. (2008). Implementing Evidence Based Practices: Strategies to 
start, sustain, and spread innovations. The National Council for Community Behavioral Healthcare 
38th Annual Conference. Boston, MA. 

Campanelli P., Cleek, A (2008). Managing Diabetes in Community Settings. New York State 
Association of Psychiatric Rehabilitation Annual Executive Seminar. Albany, NY. 

Kealey, E., Salerno, A., Margolies, P., Finnerty, M., and Cleek, A. (2008). The use of learning 
collaboratives to promote implementation of evidence-based practices in New York State. 
Presentation at the 18* Annual Conference on State Mental Health Agency Services Research, 
Program Evaluation and Policy, National Association of State Mental Health Program Directors 
Research Institute, Arlington, VA. 

Beitchman P., Campanelli P., Cleek, A. (2007). Integrating Mental and Physical Healthcare. New 
York State Association of Psychiatric Rehabilitation Annual Conference. Ehenvihe, NY. 

Cleek, A., Margolies, P., Salerno, A. (2007). Wellness Self-Management: Findings fromUIBH 
initiative and plans for a statewide rollout. New York State Association of Psychiatric Rehabilitation 
Annual Conference. Ehenvihe, NY. 
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Ades, Y., Cleek, A., Louison, A.M., Orlando, J. (2006, October). Community Based Service Models for 
Forensic Consumers. Quality in Urban Behavioral Health: from Vision to Practice Conference. New 
York, NY. 

Cleek, A., Margolies, P., Salerno, A. (2006, October). Wellness Self-Management Initiative: A Model for 
Private/ Public Collaboration. Quality in Urban Behavioral Health: from Vision to Practice Conference. 
New York, NY. 

Cleek, A., Goldberg, H, Kamnitzer, D, Nagel, L. Skoraszewski, M. (2006, June). Maximizing 
success rates for adults with chronic mental illness living in supportive housing: The implementation of 
the supported housing task force. 6* Annual New York State Supportive Housing Conference. New 
York, NY. 

Cleek, A.F. (2005, June). The Development, Structure, & Mission of UIBH: Bridging the gap between 
scientific discovery and culturally competent real world applications. 2005 Joint National Conference 
on Mental Health Block Grant & National Conference on Mental Health Statistics, Washington DC. 

Cleek, A.F. (2005, March). The Urban Institute for Behavioral Health of New York City: A model for 
Transformational Change. Poster session presented at the 2005 American College of Mental Health 
Administrators- 2005 Sante Fe Summit, Sante Fe, NM 

Cleek, A.F. (2005). Ethics in the workplace. The Institute for Community Living, Inc. Management 
Workshop, Brooklyn, NY. 

Cleek, A. F., & Merry R. (2001) Safe Havens Essentials. In Service Training, Astor Home For 
Children, Poughkeepsie, NY. 

Cleek, A. F. (2001). What to do if a parent of a child is suicidal or homicidal. In Service Training, 

Astor Home For Children, Poughkeepsie, NY. 

Cleek, A. F., & Caudill, T. (2000). The effects of domestic violence on families and children. In 
Service Training, Astor Home For Children, Poughkeepsie, NY. 

Crenshaw, D., Caudill, T., Cleek, A. F., Manning, M. K., Payne, K., & Sud, M. (2000). Difficult 
Moments in Play Therapy. New York Association for Play Therapy Annual Conference, Fishkill, NY. 

Cleek, A. F., & Price, J. L. (1999). Accord Program Needs Assessment. Presentation of data from 
multi-stage needs assessment for a high school counseling program to school and program 
administration, Louisville, KY. 

Cleek, A. F., et. al. (1998) Using Supervision Effectively in Practica. Colloquium, Spalding 
University, Louisville, KY. 

Cleek, A. F. (1998) Working in the Eamily Builders Model. Colloquium, Catholic Charities, 

Louisville, KY. 
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Micaela F. Mercado 

214 West 96* St, Apt. 4E NY, NY 10012 mm6539@nvu.edu 

EDUCATION 

May 2012 Ph.D. in Social Work, University of North Carolina at Chapel Hill 

May 2007 M.S.W., Brown School of Social Work, Washington University in St. Louis 

May 2005 B.A., Nova Southeastern University, Ft. Lauderdale, FL 

TEACHING EXPERIENCES_ 

2014 Guest Lecturer: Data Driven Performance, NYAPRS 

2013 Guest Lecturer: Managing Change in Primary & Behavioral Health Settings, NYU 
2013 Guest Lecturer: McSilver Institute for Poverty Policy and Research, NYU 
2012 Guest Lecturer: Community Based Fieldwork, Bank Street College of Education 
2012 Instructor: Evidence-based Practice Workshop, Zone 126 a Project of Rockefeller 
Philanthropy Advisors, New York, NY 

2010 Teaching Practicum: SOWO 530, Foundations of Social Welfare and Social Work, 
University of North Carolina at Chapel Hill 
2010 Guest Lecturer: SOWO 492, Seminar in Service Learning, University of North 
Carolina at Chapel Hill 

2005 Instructor: Statistical Analysis using SPSS, Nova Southeastern University, Ft. 
Lauderdale, FL 

2005 Facilitator: Psi Chi Honor Society research teams. Nova Southeastern University, 
Ft. Lauderdale, FL 


GRANTS 

New York State Health Foundation - Co-PL - $25,000 - Awarded August 2013 
Dept, of Education, Grant: 84.215P - $500,000 - Awarded Jan. 2012 

PROFESSIONAL SKILLS 


Computer Skills 

Proficient in AMOS, ATLAS, SPSS, SAS, STATA, GIS, R, Mplus, Microsoft Office, 
Crystal Reports, Tableau 


Methodological and Statistical Skills 

Research methods in social interventions 
Measurement in social intervention research 
Development of social interventions 
Propensity score analysis 
Qualitative data analysis 


Structural equation modeling 
Categorical data analysis 
Multiple regression analysis 
Multilevel mixed linear model analysis 
Latent class analysis 


Language Skills 

Proficient in spoken and written Spanish and French 
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Professional Training 

2013 Innovative Information Resources -Crystal Reports, NY 
2012 Social Solutions 2012 User Conference, NY, NY 

2012 Social Solutions, Promise Neighborhood Institute, & Policy Link, NY, NY 
2011 Center for Faculty Excellence, University of North Carolina at Chapel Hill 
2011 Teaching and Learning Symposium, University of North Carolina at Chapel Hill 


Professional Experience 


2013-Present 


2013-Present 

2011-Present 

Dec. 2011- 
2013 


2010-2011 

2008-2011 

2008-2010 

2009 

2009 

2007-2009 

2007 


Research Associate & Data Analytics Officer, McSilver Institute for Poverty Policy 
and Research, New York University, New York, NY 

• Develop, manage, and implement data collection protocols for research and 
program evaluation projects for the McSilver Institute and External Partner 
grants and contracts 

• Manage and provide professional development to undergraduate and graduate 
research assistants 

• Write federal, state, foundation, and research (NIMH, NIH, lES, DOE) grants 

• Write and submit articles to peer-reviewed journals 

• Prepare and present at national scientific conferences 

• Conduct quantitative and qualitative methods and analyses 

• Provide consultation for program implementation and data analytics 
Research Consultant, Rockefeller Philanthropy Advisors, Elmezzi Eoundation, Zone 
126 Promise Neighborhood, New York City, NY 

Research Consultant, Young Adults and Post-Secondary Education Policy Area, 
MDRC, New York City, NY 

Director of Research and Evaluation, Zone 126 a project of Rockefeller 
Philanthropy Advisors, New York, NY 

• Train and manage a team for preliminary data collection in a low-income 
community for needs assessment project 

• Identify community and student needs based on segmentation analysis 

• Develop, manage, and implement longitudinal database linking data among 25 
partner service providers serving over 3,000 children 

• Manage evaluations of community -based projects including a community grant 
program and community retreat 

• Research, develop and lead evidence-based/best-practice workshops for local 
service providers and community residents 

• Develop and manage capacity-building project to assess and report the capacity 
of service providers to host evidence-based practices 

• Develop pre- and post-assessments for service providers 

Principal Investigator, State and Institutional Policies Affecting Student Educational 
Outcomes, School of Social Work, University of North Carolina at Chapel Hill 
Principal Investigator, Sources of Social Capital Among High School Youth, School 
of Social Work, University of North Carolina at Chapel Hill 

Research Project Coordinator, CareerStart, School of Social Work, University of 
North Carolina at Chapel Hill 

Policy Research Fellow, Policy Analysis and Research: Higher Education, Jobs for 
the Euture, Washington, DC 

Field Interviewer, New Immigrant Study, Abt SRBI Inc., Washington, DC 
Certified Bilingual Field Interviewer, National Survey on Drug Use and Health, 
Durham, NC, Research Triangle Institute International 
Program Coordinator, Save the Children, La Paz, Bolivia 
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2006-2007 

2006 

2006 

2003-2005 

2002-2003 


Certified Bilingual Field Interviewer, Early Childhood Longitudinal Survey-Birth 
Cohort, St. Louis, MO, Research Triangle Institute International 

Project Evaluator, Philliber Research Associates, St. Louis, MO 

Field Interviewer, American Dream Demonstration, Center for Social Development, 

Brown School of Social Work, University of Washington in St. Louis, MO 

Certified Bilingual Field Interviewer, National Survey on Drug Use and Health, Ft. 

Lauderdale, FL, Research Triangle Institute International 

Certified Field Interviewer, One Community Partnership, University of South 

Florida, Ft. Lauderdale, FL 


Publications 

Peer-Reviewed Articles 

Small, L., Mercado, M., Gopalan, G., Pardo, G., Mellins, C., & McKay, M. Enhancing the 
emotional wellbeing of perinatally HIV infected youth across global contexts. In press. 
Global Social Welfare. 

Stem, S., Walsh, M., Mercado, M., & Lowe, E. When they call, will they come? A contextually 
responsive approach for engaging multi-stressed families in an urban child mental health 
center. In press. Research on Social Work Practice 

Woolley, M., Rose, R., & Mercado, M. (2013). Teacher teaching differently: A qualitative study 
of implementation fidelity to professional development. Journal of Education and 
Training Studies, (1), 55-68. 

Orthner, D., Jones-Sanpei, H., Rose, R., & Mercado, M., & Akos, P. (2010). CareerStart: A 

middle grades strategy for promoting student school engagement and academic success. 
Studia Sociologia, (1), 137-153. 

Orthner, D., Akos, P., Rose, R., Jones-Sanpei, H., Mercado, M., & Woolley, M. (2010). 

CareerStart: A middle-school student engagement and academic achievement program. 
Children and Schools, (32), 223-234. 

Book Chapters 

Barbarin, O., & Mercado, M., Hgjidsuren, D. (2009). Development of tolerance and respect for 
diversity in children in the context of immigration. In L. Grigorenko & R. Takanishi, 
(Eds.), Immigration, diversity, and education (pp.279-288). NY: Routledge. 

Peer-reviewed articles under review or in preparation 

Vu, C., Mercado, M., Euss., A., et al. The evolution of the Clinic Technical Assistance Center. 

In preparation. Anticipated journal - Research on Social Work Practice. 

Vu, C., Mercado, M., & Euss., et al. Addressing the gap between clinical and business practices 
through a learning collaborative model. In preparation. Anticipated journal - 
Administration in Policy in Service and Mental Health Research. 

Mercado, M. Exploring socio-cognitive and behavioral skills in the workplace: A qualitative 
study of perceived college readiness among high school students. In preparation. 
Anticipated journal - The High School Journal 

Mercado, M., Chapman, M., & Shenyang, G. Using propensity score weighting to examine the 
effects of learning communities on engagement among community college students. In 
preparation. Anticipated journal - Research in Higher Education. 

Mercado, M., Chapman, M., & Shenyang, G. A multilevel study examining first-generation 
college status, academic and social integration, and educational outcomes among 
community college students. In preparation. Anticipated journal - Teachers College 
Record. 
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Mercado, M. Advancing theory: An integrated-theoretical model for understanding the effects 
of academic and social experiences on educational outcomes among community college 
students. In preparation. Anticipated journal - Theoretical Front in Fligher Education. 

Mercado, M. The intricacies of state and institutional policies affecting student placement in 
community college courses. In preparation. Anticipated journal - Higher Education 
Management and Policy. 

Conference Presentations 

Vu, C., Mercado, M., & Fuss., A. (2014, March 3). Addressing the gap between clinical and 
Business practices through a learning collaborative model. Oral presentation, 27^’’ Annual 
Children’s Mental Health Research and Policy Conference, Tampa, FL. 

Mercado, M., Osuji, H., McKay, M. (2014, January 18). Do community HIV mentors trained in 
evidence-based, youth-focused HIV prevention program have an effect on youth outcomes ? 
Oral presentation, 18th Annual Conference of the Society for Social Work and Research, 

San Antonio, TX. 

Mercado, M., & Baharie, N. (2014, January 17). Examining the association between food 

insecurity and children's educational outcomes. Oral presentation, 18th Annual Conference 
of the Society for Social Work and Research, San Antonio, TX. 

Cleek, A., Mercado, M., Fuss, A. (2013, December 3). Using data in a transformed behavioral 
health system. Oral presentation. Office of Mental Health Children’s Mental Health 
Coaling Conference, Saratoga, NY. 

Mercado, M. (2012, March 31). A multilevel study examining first-generation status, academic 
and social integration, and educational outcomes among community colleges students .Oral 
presentation. Teachers’ College, Columbia University, New York, NY. 

Mercado, M. (2012, March 31). Examining the effects of learning communities on engagement 
among community college students. Oral presentation. Teachers’ College, Columbia 
University, New York, NY. 

Mercado, M. (2012, January 11). A multilevel study examining first-generation status, 

academic and social integration, and educational outcomes among community colleges 
students. Oral presentation, 16th Annual Conference of the Society for Social Work and 
Research, Washington, DC. 

Chapman, M. & Mercado, M. (2012, January 11). Gang activity among new immigrant Latina 
adolescents. Oral presentation, 16th Annual Conference SSWR, Washington, DC. 

Mercado, M. (2011, October 8). Examining the effects of learning communities on engagement 
among community college students. Oral presentation. National Conference on Students in 
Transition. St. Louis, MO. 

Mercado, M. (2011, March 31). Exploring socio-cognitive and behavioral skills in the 

workplace: A qualitative study of perceived college readiness among high school students. 
Oral presentation, UNC Research Conference, University of North Carolina at Chapel Hill 

Mercado, M. (2011, May 2). Examining the community college experience: The effects of 
academic and social integration on students ’ educational outcomes. Accepted Oral 
presentation. Community College Interdisciplinary Research Forum, University of 
Michigan, MI. 

Mercado, M. (2011, April 15). Understanding the community college experience: The effects of 
integration on students ’ educational outcomes. Oral presentation. Harvard Graduate School 
of Education 16* Annual Research Conference, Boston, MA. 
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Mercado, M. (2011, April 15). Examining the effects of academic and social interventions on 
community colleges students ’ engagement. Oral presentation, Harvard Graduate School of 
Education 16* Annual Research Conference, Boston, MA. 

Mercado, M. (2011, June 12) . Understanding the community college experience: The effects of 
integration on students’ educational outcomes. Oral presentation. International Conference 
on Student Success. Educational Policy Institute, San Diego, CA. 

Orthner, D., Akos, P., Rose, R., Jones-Sanpei, H., & Mercado, M. (2010, January 15). 

Relevance in the curriculum: The effect of teacher provided career examples on the math 
and reading performance of 8th grade students. Oral presentation, 14th Annual Conference 
of the Society for Social Work and Research, San Erancisco, CA. 

Orthner, D., Akos, P., Rose, R., & Mercado, M. (2010, January 15). Promoting school 

engagement and valuing: The effects of teacher-provided career examples on intermediate 
indicators of student academic success. Oral presentation, 14th Annual Conference of the 
Society for Social Work and Research, San Erancisco, CA. 

Pettus, C., Washington, T., Nwabuzor, L, & Mercado, M. (2008, October 31). The search for 
theory in social work research: A social identity perspective. Oral presentation, 54* Annual 
Council of Social Work Education Conference, Philadelphia, PA. 

Mercado, M. (2007, March). Theoretical approach: Child support savings accounts. Poster 
presentation, Eirst Annual National Eatherhood & Eamilies Conference, Eathers and 
Eamilies Coalition, Inc., Phoenix, AZ. 

Mercado, M. (2005, April). A cross-cultural perspective of cohabitation among young adults. 
Oral presentation. Student Research Symposium, Nova Southeastern University, PE. 

Silver, M., & Mercado, M. (2005, March). The dynamics of intergenerational relationship 
change in emerging adulthood. Oral presentation. Society for Research on Adolescence, 
Second Annual Conference on Emerging Adulthood, Miami, PE. 


Honors and Awards 


2007-2012 

2011 

2011 

2011 

2009 

2006-2007 

2006-2007 

2005-2007 

2005 


Royster Fellow, University of North Carolina at Chapel Hill 

Dissertation Fellowship, MDRC, New York City, NY 

Future Faculty Fellowship, University of North Carolina at Chapel Hill 

Social Science Presentation Award, University of North Carolina at Chapel Hill 

Policy Fellow, Education Pioneers, Washington, DC 

Shanti Khinduka Fellowship for International Social Work Education, Brown 
School of Social Work, Washington University in St. Louis 

Graduate Fellow, Center for Latino Lamily Research, Washington University in St. 
Louis 

Whitney M. Young Jr. Fellow, Brown School of Social Work, Washington University 
in St. Louis 

James Farquhar Award, Nova Southeastern University, Ft. Lauderdale, FL 


Reviewer _ 

Journal of International Students, 2014-Present 
The High School Journal, 2014-Present 

Global Social Welfare: Research, Policy, and Practice (GSWE), 2013-Present 
National Conference on Students in Transition, 2012-Present 





Ashley Fuss, L.M.S.W 



EDUCATION 

FORDHAM UNIVERSITY GRADUATE SCHOOL OE SOCIAL SERVICE New York, NY 
Master of Social Work, May 2011 
Concentration in Research 

Thesis: Effects of Parent-Child Discrepancies of Family Functioning on Youth Disruptive 
Behavior: A Multiple Family Group Intervention 

QUEENS COLLEGE, CITY UNIVERSITY OE NEW YORK Flushing, NY 

Bachelors of Arts Degree in Sociology, May 2008 
Major in Sociology, Minor in Journalism 
Research project: Should Plan B he sold without a prescription? 


PEER REVIEWED PUBLICATIONS 

Accepted 


1. Gopalan, G., Bannon, W., Dean-Assael, K., Fuss, A., Gardner, F., LaBarhera, B., &McKay, M. 
(2011).Multiple Family Groups: An engaging intervention for child welfare involved families. 
Child Welfare. NIHMS ID # 316860 

2. McKay, M.M., Gopalan, G., Franco, F., Dean-Assael, K., Chacko, A., Jackson, J., & Fuss, A. 
(2011). A collaboratively designed child mental health service model: Multiple Family Groups 
for urban children with conduct difficulties. Research in Social Work Practice. NIHMS ID # 
287779 

3. Gopalan, G., Alicea, S., Conover, K., Fuss, A., Gamer, F., Pardo, G., & McKay, M. (2012). 
Step-Up: Engagement strategies and attendance rates for an after school mental health program. 
Journal of Early Adolescence. 

4. Gopalan, G., Fuss, A., & Wisdom, J. (In Press). Multiple family groups to reduce child behavior 
difficulties: Influences on retention for Child Welfare-Involved Caregivers. Research on Social 
Work Practice. 

5. Stephens, T., McGuire-Schwartz, M., Rotko , F., Fuss, A., McKay, M. (In Press) A Learning 
Collaborative Supporting the Implementation of an Evidence-Informed Program, the “4Rs and 
2Ss for Children with Conduct Difficulties and their Families.” Journal of Evidence-Based Social 
Work. 






Under Review 


Gopalan, G., Small, L., Fuss, A., Bowman, M., Jackson, J., Marcus, S., McKay, M. (Under 
Review) Multiple Family Groups to reduce child behavior difficulties: Moderating effects of 
child welfare status on child outcomes. 

In Preparation 


Vu, C.M., Mercado, M., Fuss, A., Peri, D., McKay, M. (In preparation). Addressing the Gap 
between Clinical and Business Practices through a Learning Collaborative Model. 

Mercado, M., Fuss, A., Ali, S., Franco, L., Dean-Assael, K., McKay, M. (In preparation). 
Strengthening Core Competencies for Social Workers working with Veterans and their Families. 

Mercado, M., Fuss, A., Peri, D. Cleek, E., McKay, M. (In preparation). Assessing Clinic 
Finances to Improve Business Operations through Financial Management Trainings. 

Acri, M., Mercado, M., Franco, L., Dean-Assael, K., Beharie, N., Fuss, A., Cleek, A., Salerno, 
A., McKay, M. (In preparation). Developing a Clinic Technical Assistance Center during Health 
Care Reform 


PROFESSIONAL REPORTS 

Fuss, A., Mercado, M., Cleek, A. (2013). An Evaluation Report of the Leadership Development 
Institute’s Parents Leading Advocacy Now (PLAN) Training. Eunded by NAMI-NYC Metro. 

Fuss, A., Mercado, M., Cleek, A. (2013). An Evaluation of the Eamily Support Network 
Learning Collaborative. Eunded by NAMI-NYC Metro. 

Mercado, M., Fuss, A., Cleek, A., (2013) Preliminary Findings of NAMFs Peer-to-Peer Program: 
Wellness-Training Supplement Pilot. Funded by NAMI-NYC Metro. 

Mercado, M., Fuss, A., Kleinbardt, E., Ali, S. (2013) The Clinic Technical Assistance Center: 
Strategic Planning Data Book. Funded by New York State Office of Mental Health 

Beharie, N., Fuss, A., Szkola, J., Mercado, M. (2013) Presbyterian Senior Services: Data 
Summary Report. Funded by the McSilver Institute 

Gopalan ,P., Mercado, M., Fuss, A., Cleek, A. (2013) Preliminary Report of the YMCA’s Y 
Roads Program. Funded by the YMCA. 


INTERVENTION MANUAL 

Salerno, A. Cleek A., Fuss, A., Beharie, N. (2012). Knowledge Empowers You, 2"'^ Edition. 
Albany, New York, New York State Office of Mental Health. 



PROFESSIONAL PRESENTATIONS 


Completed 

Fuss, A. (2012, January). Effects of parent-child discrepancies of family functioning on youth 
disruptive behavior: A multiple family group intervention. Poster presentation for the Society for 
Social Work and Research (SSWR) 16* Annual Conference in Washington, DC. 

Fuss, A. (2012, October). “Focus on Field: Research” Presented at Fordham University, GSS, 
Seminar 

Gopalan, G., Fuss, A., Wisdom, J. (2013, March). Multiple Family Groups: Factors affecting 
retention for caregivers involved in the child welfare system. Paper presented at the 26th Annual 
Children’s Mental Health Research and Policy Conference in Tampa, FF. 

Cleek, E., Tse, J., Wofsy, M., Fuss, A. (2013, April) Accountability in a changing healthcare 
system: the role of clinical pathways, outcome measurement and decision support. Presented at 
the Preconference University Advances in Clinical Treatment and Services for Children at The 
National Council for Behavioral Health Conference in Las Vegas, NV. 

Cleek, A., Mercado, M., Fuss, A. (2013, December) Using Data in a Transformed Behavioral 
Health System. Presented at the Children’s Mental Health Services Staff Development Training, 
Forum Pathways to Success: Managing Total Wellness in Saratoga, NY. 

Cleek, A., Beharie, N., Fuss, A. Salerno, A., (2013, December) Knowledge Empowers You 
(KEY): Empowering Youth to Make Decisions that Work for Them. Presented at the Children’s 
Mental Health Services Staff Development Training Forum, Pathways to Success: Managing 
Total Wellness in Saratoga, NY. 

RESEARCH EXPERIENCE 

2012-Present NEW YORK UNIVERSITY New York, NY 

SILVER SCHOOL OE SOCIAL WORK 

MCSILVER INSTITUTE EOR POVERTY, POLICY AND RESEARCH 
Junior Research Scientist (Director: Dr. Mary McKay) 

• Contribute to NIMH funded research projects focused on poverty-impacted youth, 
families and communities 

• Analyze data to examine the impact of family and youth outcomes associated with 
treatment interventions 

• Examine the impact of various technical assistance and training programs disseminated 
from McSilver 

• Prepare federal and private grant submissions 

• Partner with community-based organizations to implement evaluation protocols within 
their organizations 

• Design program evaluation and research assessment batteries 

• Train staff on the collection and use of data 

• Prepare data and evaluation reports to partners and funders 

• Prepare manuscripts for publication 

• Co-design a staff development training for mental health providers working with 
veterans 




Conduct focus groups with staff and consumers from community-based organizations 


2012-Present 


2012-Present 


2011-2013 


2011-2012 


2011-2011 


THE INSTITUTE EOR COMMUNITY LIVING, INC. New York, NY 

SUPPORTIVE SERVICES EOR VETERAN EAMILIES PROGRAM (SSVE) 
Evaluation Specialist (Supervisor: Dr. Elizabeth Cleek) 

• Analyze outcome data to examine the impact the SSVF program has on individuals 
enrolled in the program 

• Co-develop an evaluation work plan for the SSVF program 

• Prepare evaluation forms for program use 

• Develop program database and data dictionaries 

UNITED NEIGHBORHOOD HOUSE New York, NY 

SILBERMAN SCHOOL OE SOCIAL WORK AT HUNTER COLLEGE 
Research Analyst (Principal Investigator: Dr. Mimi Abramovitz) 

• Conduct statistical analysis to examine members’ perceptions of settlement houses 

• Data entry, coding and management 

NEW YORK UNIVERSITY New York, NY 

SILVER SCHOOL OE SOCIAL WORK 

Research Analyst (Principal Investigator: Dr. Geetha Gopalan) 

• Prepare federal grant application for submission 

• Qualitative interview coding 

• Complete reliability checking 

• Quantitative data analysis 

• Prepare manuscripts for publication 

THE URBAN INSTITUTE EOR BEHAVIORAL HEALTH New York, NY 

Research Project Coordinator (Director: Dr. Andrew Cleek) 

• Downstate coordination for the New York State Clinic Technical Assistance Center 

• Dissemination and project management of CTAC initiatives 

• Develop and maintain databases 

• Data management and data analysis 

• Co-author Knowledge Empowers You (KEY) Workbook 

• Manage the implementation of KEY in community-based settings 

• Geo-Mapping 

• Management and submission of network data for New York State Department of 
Health (NYSDOH) Health Homes’ project 


MOUNT SINAI SCHOOL OE MEDICINE New York, NY 

DEPARTMENT OE PSYCHIATRY AND COMMUNITY MEDICINE 
Research Assistant (Principal Investigator: Dr. Geetha Gopalan) 

• Provide support for overall research project management 

• Prepare recruitment materials for research study 

• Co-design the qualitative Interview Guide 

• Conduct qualitative interviews 

• Qualitative interview coding 

• Complete reliability checking 



• Data entry and analysis 

2011 2011 CHILD WELFARE ORGANIZTION PROJECT New York, 

NY 

Research Assistant (Principal Investigator: Dr. Geetha Gopalan) 

• Prepare assessment batteries for children and caregivers 

• Administer assessment measures to children and caregivers 

• Develop database for assessments to be entered 

• Data entry and cleaning 

MOUNT SINAI SCHOOL OF MEDICINE New York, 

DEPARTMENT OF PSYCHIATRY AND COMMUNITY MEDICINE 
Research Social Work Intern (Director: Dr. Mary McKay) 

• Contribute to research projects focused on low-income families with mental 
health difficulties 

• Provide support for overall research project management 

• Administer research- based assessments 

• Prepare manuscripts for publication 

• Coordinated follow-up events for research participants 

• Prepare Interim Report to the Robinhood Foundation 

• Analyze outcome data, prepare databases, data cleaning, entry and coding 

• Conduct home visits 

• Co-facilitate a Multiple Family Group for children with behavioral problems 

PROFESSIONAL CLINICAL EXPERIENCE 

2010 2011 CHILD WELFARE ORGANIZTION PROJECT New York, 

NY 

Social Work Intern 

• Co-facilitate a family-based group for children who were reunited with their 
families from foster care 

2009-2010 AFTER-SCHOOL PROGRAM AT P.S. 86 Jamaica, NY 

QUEENS COMMUNITY HOUSE 
Social Work Intern 

• Facilitate social skills activity groups for youth grades 3-6 

• Provide individual supportive counseling 

• Plan staff development activities, 

• Administer program and service evaluation surveys 

• Parent outreach activities 

VOLUNTEER EXPERIENCE 

2012-2013 THE SOLDIERS 

PROJECT LONG 

ISLAND CHAPTER 

Community Volunteer 


2010-2011 

NY 


6 



CERTIFICATION AND PROFESSIONAL AFFILIATION 


Society for Social Work and Research 
New York State Licensed Social 
Worker Veterans Mental Health 
Coalition 


7 



APPENDIX C 
References for RFP 


Alemi F, Erdman H, Griva I, Evans CH (2009). Improved statistical methods are needed to 
advance personalized medicine. Open Translational Medicine Journal, January 1; 1:16-20. 
doi:10.2174/1876399500901010016 

Allegheny County Integrated Children's Services Plan: Fiscal Year 2012-2013. October, 2011. 

Allegheny County Mental Health Plan for Adults, Older Adults and Transition-Age Youth with 
Serious Mental Illness and Co-occurring Disorders: Fiscal Year 2013-2014. May, 2012. 

Barrett JP (1974). The co-efficient of determination. The American Statistician, 28(1): 19-20. 

Begle AM, Dumas JE, Hanson RF (2010). Predicting child abuse potential: an empirical 
investigation of two theoretical frameworks. Journal of Clinical Child and Adolescent 
Psychology. 39(2): 208-19. 

Cherna M (2014). Presentation: Blue Ribbon Commission on Child Protection, Los Angeles 
County. March 10. 

Cicchetti D, Blender JA (2006). A multiple-levels-of-analysis perspective on resilience: 
implications for the developing brain, neural plasticity, and preventive interventions. Annals of 
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3. BUDGET AND BUDGET DESCRIPTION 

Budget Narrative/Description 

1. There are no anticipated revenues as shown in Line Item 1 - Budget Revenue. 

2. Line Item 2 outlines budget expenses including personal services and associated agency 

costs. There will be three staff from Coordinated Care Services, Inc. (CCSI) included on the 

project and three from NYU-McSilver, described as follows: 

• John Crilly, PhD, MPH, MSW : Principle Investigator. Dr. Crilly's role will be to oversee the 
entire project. Although he will work closely with Dr. Cleek in these areas, he will be 
responsible for overseeing the analysis of qualitative data, development and analyses of the 
predictive model, and all interpretations resulting therefrom. He will be on site to 
participate in the focus and design groups and assist in the cross-walk collaborations both 
with NYU and the software developers. 

TBD : Project Coordinator. The CCSI Project Coordinator will be a Masters-trained, senior- 
level CCSI employee who has experience in the arenas of direct care, program evaluation, 
and research with a strong understanding of data and technology. This PC will be 
responsible for interactions with AC DHS staff to plan logistics in the AC DHS environment 
and develop and implement focus groups. This PC will run all focus groups and will be 
responsible for the recording and analysis of the qualitative data in conjunction with Dr. 
Crilly. This PC will be responsible for maintaining project adherence to the timeline. 

TBD : Data Analyst. The data analyst will be a Masters-trained Sr. Programmer/Analyst who 
will be proficient in the statistical software and analyses methods necessary to conduct the 
work in this proposal. This will include SPSS, SAS, Stata, R, and Atlas (qualitative data 
analysis software). 

NYU-McSilver staff will include: 

• Andrew Cleek, PsyD : Co-Investigator. Dr. Cleek's role will be to oversee the Project as it 
occurs at the NYU-McSilver site. Dr. Cleek will assist in gathering experts for input into the 
predictor and outcome variable designations and expediting the process. He will 
collaborate with Dr. Crilly in the development, analysis, and interpretation of the predictive 
models. He will also assure smooth transitions of processes requiring crosswalks between 
NYU-McSilver and CCSI, AC DHS, and software developers with Dr. Crilly. 

• Micaela Mercado, PhD, LMSW : Evaluation Staff. Dr. Mercado will facilitate and oversee the 
operation of the Predictor/Outcome variable determination at NYU. Working with Dr. 

Cleek, she will design the appropriate group formats and recruit panels of experts to 
provide input into the features of Questions A and B in this proposal. 

• Ashley Fuss, MSW : Project Coordinator. Ms. Fuss will be responsible for the operation of 
the project for NYU-McSilver. She will work closely with CCSI and AC DHS and will 
coordinate all aspects of the project at NYU-McSilver. She will be responsible for ensuring 
adherence to the timeline. She will also be involved as a note-taking participant in the AC 
DHS focus groups to ensure an adequate link between the work conducted by Dr. Mercado 
and the input provided by the focus group members. 

3. Each staff member is individually listed with their fully loaded compensation rate which 

includes costs associated with project time, agency administrative time, vacation and holiday 
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time including mandated and non-mandated benefits as well as any associated office supplies, 
office equipment, infrastructure and agency administration and overhead. 

4. An estimation of their time required for this project is listed under the FTE column, which is 
the Full Time Equivalent based on a 40 hour work week. Work tasks included in this estimation 
are outlined above. 

5. The Extended Price column is calculated by multiplying the fully loaded compensation rate 
by the FTE. The Line Item 2 Subtotal is $317,925. 

6. Line Item 3 summarizes travel and associated costs. Mileage and related ground 
transportation expenses are calculated along with per diem hotel and meal related expenses. It 
is anticipated that the Principal and Co-Investigator will travel to Allegheny County on a bi¬ 
monthly basis, 2.5 days per trip. Trip duties will include attending focus group meetings, 
meetings and consultation with AC DHS staff, ensuring overall operation of the project and, 
later, meeting with software developers. The CCSI and NYU Coordinators will have a total of 14 
trips, 2.5 days per trip, to be distributed throughout the year as needed for conducting focus 
groups and attending meetings with AC DHS staff at all levels. Mileage was calculated using 
round trip miles and $0.56/per mile mileage reimbursement. The US General Services 
Administration figures were used to calculate per diem travel costs. Greater detail for each 
staff member is listed below. The Line Item 3 Subtotal is $38,228.00. 


Mileage and other travel 

Total 

1 staff NYU: 14 trips for 2.5 days each 


Co-lnv NYU: 1 time bi-monthly for 2.5 days 


1 staff CCSI: 14 trips for 2.5 days each 


Princ Inv CCSI: 1 time bi-monthly for 2.5 days 


Total Mileage and other travel 

$ 


Hotel, meals, etc. 

Total 

1 staff NYU: 14 trips for 2.5 days, 2.5 nights 


Co-lnv NYU: 1 time bi-monthly for 2.5 days 


1 staff CCSI : 14 trips for 2.5 days, 2.5 nights 


Princ Inv CCSI: 1 time bi-monthly for 2.5 days 


Total Hotel, meals, etc. 

$ 
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APPENDIX A 

PROPOSAL AUTHENTIFICATION FORM 
TITLE: Decision Support Tools and Predictive Analytics in Human Services 

DUE DATE: APRIL 18, 2014 


The undersigned hereby offers to furnish and deliver the services as specified in strict accordance with the 
RFP and scope of proposal, all of which are made a part of this request. This offer is not subject to 
withdrawal without permission of the County of Allegheny Department of Human Services Director. 


FULL LEGAL COMPANY NAME 


; ^o<5 . “XyrNt 


DOING BUSINESS AS: 


S>Gc\fY\€- 


STREET ADDRESS: ^V<jvQ 


CITY, STATE AND ZIP CODE:. 


AUTHORIZED SIGNATURE: 






PRINT NAME:_ TG ^S Q<\ _ 

TITLE OF AUTHORIZED SIGNER: C 


TELEPHONE#: 5^^ ' S\^C> faX#: 


WEBSITE URL: 


E-MAIL ADDRESS: 


>u. 3U3 'd cs \ . o 

. e.r< 


Chapter 2 

COMPANY INFORMATION 

(This information is for tracking purposes only and has no role in the determination of the responsible proposer.) 

□ Check here if your firm is registered with the Allegheny County Department of Minority, Women and 
Disadvantaged Business Enterprises 

□ Check here if your firm is a "Minority Business Enterprise" or "MBE" as defined in the Small Business 
Act, 15 use 

□ Check here if your firm is a "Women Business Enterprise" or "WBE" as defined in the Small Business 
Act, 15 use 

□ Check here if your firm is a "Small Business" as defined by the Small Business Administration (13 C.F.R. 
121.201, in most cases, this means a business with 500 or fewer employees) 

NOTE: THIS PAGE MUST BE SUBMITTED WITH YOUR PROPOSAL ALL PAGES REQUIRES A LIVE 

SIGNATURE SIGNED IN BLUE INK. 
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Documentation to be attached to 
MA/V/DBE Participation Waiver Request 

Business Practices and Operations Profiie 

Coordinated Care Services, Inc. (CCSI) is a not-for-profit (501 c3) management services firm with a long 
history of providing support to organizations in the behavioral health and human services fields so that 
they can improve services for those in need. 

CCSI is a team of talented, highly motivated professionals with specialized expertise and an understanding of 
best and evidence-based practices, coupled with the ability to develop and implement practical solutions that 
work within our customers’ environments. 

• Our core service lines, which are offered both individually and in combination based on customer 
needs, include: 

• Contract and Financial Management 

• Service Quality and System Development 

• Evaluation and Services Research 

• Cultural and Linguistic Competence 

• Program and Project Management. 

• CCSI Mission Statement 

CCSI is a management services organization that provides integrated solutions to health and 
human services customers to fulfill their missions, achieve their goals, and improve the lives of 
those they serve. 

• CCSI Values 

All that we do at CCSI is guided by the following values: 

o Customer Focused: Understanding and meeting the needs of both internal and external 
customers, being responsive. 

o Solutions Focused: Providing the best possible answer to problems and needs; thinking 
comprehensively and cross-functionally; entrepreneurial, 
o Collaborative: Working together to achieve goals and objectives; partnering with clients; 
ensuring win/win relationships. 

o Quality: Excellence in all that we do and how we do it (people, processes, and systems); 

clear, concise, timely, accurate, and outcome-oriented, 
o Fiscally Responsible: Financial accountability and sustainability; understanding and 
managing pricing, costs, funding, etc.; providing a solid ROI on the client’s investment. 

• Statement of Strategic Direction 

CCSI will ensure growth and sustainability and pursue its mission by: 
o Building organizational capability and capacity 
o Increasing organizational alignment, collaboration and teamwork 
o Achieving operational excellence 
o Improving financial operating performance 

Diversity at CCSI 

CCSI recognizes the importance of a culture that embraces diversity and appreciates, understands and 
values individual differences. We are committed to increasing the diversity of our staff at all levels; to creating 
an inclusive, respectful and equitable environment; to serving our diverse customers with culturally sensitive 
services; and to continually improving CCSI’s culture through leadership, policies and practices. 

Explanation for MA/V/DBE Waiver Request 

CCSI is Requesting a MA/V/DBE waiver because are not designated as a women owned business. CCSI 
values women and minorities in the work place, and two of the company’s three officers are female; 

President, Anne Wilder, and Sr. Vice President & Chief Operating Officer, Donna Peri. 76% of all CCSI 
employees are female. 22% of CCSI employees associate themselves as a minority. 
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APPENDIX D 


Allegheny County 

Vendor Creation Form 


I I Add O Change Supplier No. 


Controller's use only: 

Supplier No._ 

1099 Eligibility: []] Yes Q No 


Company Information: 


Federal Tax ID (TIN) 


Co6rAv\rNcA«?A (IcyCe "COvC. 

Company Name (Please type or print) 


Required information 
Type of Service Provided 


^^Independent Contractor 

EH Rent 

1 1 Maintenance/Service Agreement 

1 1 Care Giver 

1 1 Insurance 

□ Legal 

r~l Personal Reimbursement 

1 1 Medical 

EH Other (please list) 



3 a - 


Original W-9 must be attached 


Type of Commodity Provided 

(please describe below) 





Required Information 

Minority Or Women Owned EH Yes PTno 

If yes select ethnicity and gender of the vendor below: 

I I Asian Pacific American 

I I Black American 

I I Hispanic American 

I I Native American 

I I Subcontinent Asian American 

I I Non-Minority Woman 

I I Other 

If Yes I I Male I I Female 

Certified By: EH PAUCP EH PADGS EH Non PA Certification 

(attach copy of certification) 

Non-Profits including Faith Based Organizations 

EH Faith Based Non-Minority 
EH Faith Based Minority 
EH African American Non-Profit 
[/f Other Non-Profit 

EH Yes No 


Outreach Manager Interface 






APPENDIX D 

Industry Classification by NAICS Code 
Primary Industry 

Secondary Industry (if applicable) 


*lf code is not known go to http://www.census.gov/epcd/naics02/naicod02.htm and seiect the correct code. 



Required Information 

Supplier/Remit To Information (Search Type "V") - (Where check will be mailed for payment. Check must be 
made payable to exact name listed under TIN provided or check cannot be processed.) 

Please print or type 


Supplier/Payee Name 
Address Line 1 
Address Line 2 




Address Line 3 


City 




ZIP Code 




Telephone Number 




Fax Number 

*lf the "remit to" information provided on form does not match invoices submitted for payment, the Controller's 
Office MUST contact supplier to verify address information before payments are processed. Thank you for your 
cooperation. 
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If the Allegheny County Department with which you do business is known, providing the information 
below will help in the processing of your payments. Failure to include the information may result in 
processing delays. 


Allegheny County 
Departmental Contact 

Name Leslie Lewis-Pollard 

Telephone 

No. 412-350-5663 

Fax No. 412-350-3414 



EMail 

Address: 


Llewis- 

pollard@alleghenycounty.us 


Email Address: 


^ txA\ 


(LCJS, \ .OC( 




Form W-9 
(Rev. January 2011) 
Department of the Treasury 
Internal Revenue Service 


Request for Taxpayer 
Identification Number and Certification 


Give Form to the 
requester. Do not 
send to the IRS. 


Name (as shown on your income tax return) 


o 

a 




o 

c 

'u 

a 


CM 

0 ) 

O) 

OJ 

a 

c 

o 

(0 

c 

o 


E 

In 

c 


o 

4) 

a 

(0 

0) 

o 

CO 


Coordinated Care Services, Inc. _ 

Business name/disregarded entity name, if different from above 


Check appropriate box for federal tax 

classification (required): [] Individual/sole proprietor [3 C Corporation Q S Corporation 

n Partnership O Trust/estate 


l~1 Limited liability company. Enter the tax classification (C=C corporation, S=S corporation, P=partnership) ► 

1_1 Exempt payee 

r~l Other (see instructions) ► 



Address (number, street, and apt. or suite no.) 

1099 Jay Street, Bldg J, 3rd Fir 

Requester’s name and address (optional) 

City, state, and ZIP code 

Rochester, NY 14611 




List account number(s) here (optional) 


Part I 


Taxpayer Identification Number (TIN) 


Enter your TIN in the appropriate box. The TIN provided must match the name given on the “Name” line 
to avoid backup withholding. For individuals, this is your social security number (SSN). However, for a 
resident alien, sole proprietor, or disregarded entity, see the Part I instructions on page 3. For other 
entities, it is your employer identification number (EIN). If you do not have a number, see How to get a 
TIN on page 3. 


Note. If the account is in more than one name, see the chart on page 4 for guidelines on whose 
number to enter. 


Part II 


Certification 


Under penalties of perjury, I certify that: 


1 Social security number 




- 



- 







1. The number shown on this form is my correct taxpayer identification number (or I am waiting for a number to be issued to me), and 


2. I am not subject to backup withholding because: (a) I am exempt from backup withholding, or (b) I have not been notified by the Internal Revenue 
Service (IRS) that I am subject to backup withholding as a result of a failure to report all interest or dividends, or (c) the IRS has notified me that I am 
no longer subject to backup withholding, and 


3. I am a U.S. citizen or other U.S. person (defined below). 


Certification instructions. You must cross out item 2 above if you have been notified by the IRS that you are currently subject to backup withholding 
because you have failed to report all interest and dividends on your tax return. For real estate transactions, item 2 does not apply. For mortgage 
interest paid, acquisition or abandonment of secured property, cancellation of debt, contributions to an individual retirement arrangement (IRA), and 
generally, payments other than interesKand dividends, you are not required to sign the certification, but you must provide your correct TIN. See the 
instructions on page 4._tT ______ 



Signature of t 

U.S. person ► 

^ t -Date^ 

d 


General Instruction^' 

Section references are to the In^ 
noted. 

i Note. If a requester gives you a form other than Form W-9 to request 

\ your TIN, you must use the requester’s form if it is substantially similar 

jjnal Revenue Code unless otherwise to this Form W-9. 

Definition of a U.S. person. For federal tax purposes, you are 


PurpOS© of Form considered a U.S. person if you are: 


A person who is required to file an information return with the IRS must 
obtain your correct taxpayer identification number (TIN) to report, for 
example, income paid to you. real estate transactions, mortgage interest 
you paid, acquisition or abandonment of secured property, cancellation 
of debt, or contributions you made to an IRA. 

Use Form W-9 only if you are a U.S. person (including a resident 
alien), to provide your correct TIN to the person requesting it (the 
requester) and, when applicable, to: 

1. Certify that the TIN you are giving is correct (or you are waiting for a 
number to be issued), 

2. Certify that you are not subject to backup withholding, or 

3. Claim exemption from backup withholding if you are a U.S. exempt 
payee. If applicable, you are also certifying that as a U.S. person, your 
allocable share of any partnership income from a U.S. trade or business 
is not subject to the withholding tax on foreign partners’ share of 
effectively connected income. 


• An individual who is a U.S. citizen or U.S. resident alien. 

• A partnership, corporation, company, or association created or 
organized in the United States or under the laws of the United States, 

• An estate (other than a foreign estate), or 

• A domestic trust (as defined in Regulations section 301.7701 -7). 
Special rules for partnerships. Partnerships that conduct a trade or 
business in the United States are generally required to pay a withholding 
tax on any foreign partners’ share of income from such business. 
Further, in certain cases where a Form W-9 has not been received, a 
partnership is required to presume that a partner is a foreign person, 
and pay the withholding tax. Therefore, if you are a U.S. person that is a 
partner in a partnership conducting a trade or business in the United 
States, provide Form W-9 to the partnership to establish your U.S. 
status and avoid withholding on your share of partnership income. 
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